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AN  ANALYSIS  OF 

EARNED  VALUE  MANAGEMENT  IMPLEMENTATION 
WITHIN  THE  F-22  SYSTEM  PROGRAM  OFFICE’S 
SOFTWARE  DEVELOPMENT 

ABSTRACT 


Department  of  Defense  (DoD)  use  of  Earned  Value  Management  (EVM)  program 
control  tool  has  significantly  increased  in  the  last  ten  years.  DoD  acquisition  policy  and 
training  promotes  EVM  as  a  cost  and  schedule  management  tool,  tracking  the  earned 
value  of  the  work  completed  per  the  baseline  plan.  Acquisition  Category  ID  programs 
like  the  US  Air  Eorce  E-22  fighter  program  use  EVM  to  manage  their  software 
development  efforts,  but  has  the  program’s  implementation  of  EVM  followed  the 
industry-recognized  32  criteria  outlined  in  ANSI/EIA-748-A-1998  (Earned  Value 
Management  System  Standards)  necessary  to  successfully  implement  EVM? 

Using  these  32  criteria,  an  evaluation  was  performed,  aimed  at  assessing  the 
implementation  of  EVM  in  the  E-22  program.  The  goal;  to  academically  appraise  the 
program’s  use  of  EVM  in  managing  Spiral  2,  an  E-22  avionics  software  modernization 
effort.  To  accomplish  this  goal  a  detailed  evaluation  of  how  the  program  meets  the  32 
criteria  was  conducted  along  with  analysis  of  program  data,  interviews  of  subject  matter 
experts  and  a  statistical  questionnaire  conducted  with  E-22  personnel.  Results  indicated 
areas  of  possible  improvement  in  the  use  of  EVM  and  potential  changes  to  the  E-22 
development  environment  to  improve  planning,  scheduling  and  budgeting  of  the  EVM 
baseline. 
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EXECUTIVE  SUMMARY 


Department  of  Defense  (DoD)  use  of  Earned  Value  Management  (EVM)  program 
control  tool  has  significantly  increased  in  the  last  ten  years.  DoD  acquisition  policy  and 
training  promotes  EVM  as  a  cost  and  schedule  management  tool,  tracking  the  earned 
value  of  the  work  completed  per  the  baseline  plan.  Acquisition  Category  ID  programs 
like  the  US  Air  Eorce  E-22  fighter  program  use  EVM  to  manage  their  software 
development  efforts,  but  has  the  program’s  implementation  of  EVM  followed  the 
industry-recognized  32  criteria  outlined  in  ANSI/EIA-748-A-1998  Earned  Value 
Management  System  Standards  and  necessary  to  successfully  implement  EVM. 

Using  these  32  criteria,  an  evaluation  was  performed  aimed  at  assessing  the 
implementation  of  EVM  in  the  E-22  program.  The  goal:  to  academically  appraise  the 
program’s  use  of  EVM  in  managing  Spiral  2,  an  E-22  avionics  software  modernization 
effort.  To  accomplish  this  goal  a  detailed  evaluation  of  how  the  program  meets  the  32 
criteria  was  conducted  along  with  analysis  of  program  data,  interviews  of  subject  matter 
experts  and  a  statistical  questionnaire  conducted  with  E-22  personnel.  Results  indicated 
areas  of  possible  improvement  in  the  use  of  EVM  and  potential  changes  to  the  E-22 
development  environment  to  improve  planning,  scheduling  and  budgeting  of  an  EVM 
baseline.  A  concise  description  of  the  results  for  each  of  these  assessments  follows 
below.  1 

The  subjective  evaluation  of  the  32  criteria  exhibited  several  areas  of  interest. 
Eirst,  22  of  the  32  data  points  (68.75  percent)  either  met  or  exceeded  the  intent  of  their 
respective  ANSI  criterion  by  earning  an  excellent  or  satisfactory  rating.  Of  the  remaining 
data  points,  six  (18.75  percent)  exhibited  either  a  marginal  or  insufficient  rating,  while 
four  (12.50  percent)  exhibited  an  inconclusive  rating  (an  inconclusive  rating  resulted  from 
insufficient  supporting  data).  Major  factors  contributing  to  less-than-satisfactory  ratings 
included  usage  of  undefmitized  contracts  and  an  observed  lag  between  cost  data 
collection  and  reporting. 


1  Please  refer  to  Chapters  III  and  IV  for  a  more  detailed  discussion  of  these  findings. 

1 


The  questionnaire,  whieh  reaehed  100  pereent  of  government  and  eontraetor 
personnel  dedieated  to  Spiral  2  (the  first  F-22  avionics  software  upgrade  since  Initial 
Operational  Capability  (IOC)),  exhibited  the  following: 

•  Personnel  had  a  slight  majority  opinion  EVM  has  some  value 

•  Personnel  had  a  slight  majority  opinion  EVM  has  some  usefulness 

•  Most  personnel  are  not  aware  of  the  ANSI/EIA-748-A-1998  EVMS  32 

Criteria 

Additional  statistical  analysis  of  the  questionnaire  concluded  that: 

•  Higher  EVM  value  resulted  in  more  EVM  usage 

•  Higher  EVM  usefulness  resulted  in  more  usage  of  EVM  in  managing 

•  Higher  EVM  knowledge  did  not  result  in  higher  EVM  value 

Einally,  the  interviews  independently  verified  the  separate  assessment  of  the  32 
criteria  and  served  to  explain  the  results  of  the  questionnaire. 

The  summation  of  the  32  criteria  evaluation,  questionnaire  analysis,  and 
interviews  led  to  the  conclusion  that  the  E-22  Program  Team,  with  respect  to  its  software 
development  efforts,  did  meet  the  intent  behind  the  industry  standard  ANSI/EIA-748-A- 
1998  with  respect  to  most  of  the  criteria.  There  were,  however,  several  criteria  that  were 
found  to  be  marginal  or  insufficient.  Research  identified  several  areas  of  incompatibility 
with  regard  to  EVM  and  software  development  (or  any  other  similarly  dynamic 
environment)  that  may  preclude  functional  managerial  controls.  Specifically,  as  software 
development  efforts  progress  beyond  early  stages,  less-defined  tasks  become  more 
difficult  to  manage  via  EVM. 
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I.  INTRODUCTION 


A.  BACKGROUND 

1.  Earned  Value  Management 

DoD  Program  Managers’  use  of  EVM  as  a  measure  of  program  eost  and  sehedule 
performanee  has  signifieantly  inereased  in  the  last  ten  years.  DoD  acquisition  policy  and 
training  promotes  EVM  as  a  tool  for  measuring  program  health  by  tracking  the  “earned 
value”  of  the  work  completed  per  the  baseline  plan. 

Eacilitating  an  intelligent  discussion  concerning  EVM  first  requires  a  rudimentary 
knowledge  of  the  EVMS  as  it  exists  today.  Consequently,  this  paper  offers  the  following 
primer  for  those  either  new  to  or  requiring  a  refresher  in  the  basics  of  EVM.  Proposed 
methods  for  evaluating  an  EVMS  and  a  discussion  of  the  ANSI/EIA-748-A-1998  EVMS 
criteria  follow  the  primer. 


a.  An  EVM  Primer 

Earned  Value  Management;  “...a  tool  for  effectively  integrating  cost, 
schedule  and  technical  performance  managemenf’  (DAU,  2005).  To  integrate  said  cost, 
schedule,  and  performance  involves  making  those  measurements  visible.  The  core  of  that 
visibility  revolves  around  three  measurements  (DAU,  2005): 

•  Budgeted  Cost  of  Work  Scheduled  (BCWS)  -  This  measurement  sums 
the  budgets  for  all  work  scheduled  for  accomplishment — including  in- 
process  work — plus  the  amount  of  apportioned  effort  scheduled  for 
accomplishment  at  a  specified  point  in  time.  The  BCWS  value  at  project 
conclusion  equates  to  Budget  at  Completion  (BAC).  Alternate 
terminology  includes  “planned  value”  and  the  Performance  Measurement 
Baseline  (PMB). 

•  Budgeted  Cost  of  Work  Perforuied  (BCWP)  -  This  measurement 
provides  the  value  of  work  actually  performed  and  uses  budgeted  costs  to 
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calculate  the  eost  at  a  speeific  point  in  time.  Also  known  as  Earned  Value 
(EV). 

•  Actual  Cost  of  Work  Performed  (ACWP)  -  This  measurement  provides 
the  eosts  aetually  ineurred  and  reeorded  (as  opposed  to  budgeted  eosts)  in 
aceomplishing  the  work  performed  at  a  speeifie  point  in  time;  normally 
the  eontraeted  organization  provides  ACWP  data  direetly.  Often  it  is 
simply  ealled  Actual  Cost  (AC). 

Program  Managers  use  eomparisons  of  these  three  measurements  to  gauge 
a  eontraetor’s  progress  against  an  initially  agreed-upon  baseline  (as  a  rule,  the  PMB). 
For  instanee,  a  quiek  comparison  of  BCWS  and  BCWP  provides  a  useful  measurement  of 
Sehedule  Varianee  (SV).  An  intuitive  analysis  reveals  that  when  work  performed 
exeeeds  work  seheduled  for  a  given  program,  the  program  in  question  is  ahead  of 
sehedule. 

Likewise,  a  quiek  eomparison  of  BCWP  and  ACWP  provides  an  equally 
useful  measurement  known  as  Cost  Varianee  (CV).  At  an  elementary  level,  when  aetual 
eosts  exeeed  budgeted  eosts  within  a  scrutinized  program,  that  program  warrants  a  “eost 
overrun”  elassification.  Armed  with  CV  and  SV,  a  program  manager  and  his  or  her 
Integrated  Produet  Team  (IPT)  ean  now  ealculate  a  rudimentary  Estimate  at  Completion 
(EAC). 

For  those  not  familiar  with  the  term,  EAC  answers  “What  do  we  now 
expeet  the  total  job  to  eost?”  (Haupt,  2002).  Take  an  original  EAC  of  20  and  a  one  time 
CV  of  -4.  Subtraet  this  4  point  overrun  from  the  original  EAC  of  20  (i.e.,  20  -  (-4)),  and 
the  new  EAC  reflects  a  projected  eost  of  24.  The  basies  of  EAC  now  explained,  the  next 
level  of  analysis  involves  ereating  indiees  for  both  eost  and  performanee  measurements. 

Within  the  EVM  body  of  knowledge,  these  indiees  are  known  as  Cost  and 
Schedule  Performance  Index  (CPI  and  SPI).4  The  CPI  indieates  eost  performanee 


2  SV  =  BCWP  -  BCWS 

3  CV  =  BCWP  -  ACWP 

4  CPI  =  BCWP  /  ACWP 
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efficiency  related  to  work  the  contractor  has  actually  accomplished  at  a  specific  point  in 
time.  In  other  words,  it  provides  a  measurement  of  the  value  of  work  the  program 
receives  from  every  dollar  given  towards  the  effort.  For  example,  assume  BCWP  =  8  and 
ACWP  =  10.  The  resulting  CPI  (.8)  reports  that  every  dollar  invested  into  the  project 
results  in  .80  cents  of  effort.  CPI’s  ultimate  use  (assuming  a  constant  CPI)  stems  from 
enabling  government  IPTs  to  project  the  final  cost  of  a  contract  and  even  determine  the 
likelihood  that  the  contractor  can  recover  (Heise,  1991,  p.  95). 

Along  those  same  lines,  SPI  indicates  schedule  efficiency  at  a  specified 
point  in  time.  For  example,  if  BCWP  =  BCWS,  then  SPI  =  1.  An  index  of  1  indicates 
that  the  supplier  is  performing  on  schedule  whereas  an  index  of  1 . 1  indicates  an  ahead-of- 
schedule  condition  (Smith,  1977).  Like  CPI,  SPI’s  ultimate  utility  stems  from  bestowing 
the  ability  to  project  the  final  completion  date  of  a  contract  alongside  the  probability  that 
the  contractor  can  meet  or  beat  the  original  project  completion  date,  given  a  reliable 
budget. 

This  segment  represented  a  very  basic  working  knowledge  of  EVM;  the 
next  topic  concerns  evaluating  the  effectiveness  of  EVM  within  a  given  program. 

b.  Evaluating  the  Effectiveness  of  an  EVMS 

When,  in  1995,  the  National  Defense  Industrial  Association  (NDIA) 
decided  it  was  too  unwieldy  and  expensive  to  abide  by  DoD’s  35  Cost/Schedule  Control 
Systems  Criteria  (C/SCSC),  industry  leaders  took  initiative  and  developed  EVMS,  which 
remains  more  or  less  in  its  same  form  today. ^  Eittle  more  than  twenty  months  later,  the 
32  guidelines  from  the  new  industry  standard,  American  National  Standards  Institute  / 
Electronic  Industries  Association  (ANSI/EIA)  748,  became  the  DoD  baseline  for  EVMS, 
as  well.  Essentially,  ANSI/EIA-748-A-1998  states  how  to  effectively  apply  earned  value 
concepts  that  will  aid  in  successful  program  management  (Eleming  &  Koppelman,  2000). 
Even  knowing  that,  one  might  ask,  “So  what?” 

SPI  =  BCWP  /  BCWS 
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To  begin  with,  history  has  shown  that  no  single  EVMS  ean  hope  to  meet 
every  need  (management,  reporting,  ete.)  with  respeet  to  performanee  measurements. 
Differenees  in  programs,  as  varied  and  unrelated  as  organizations,  weapon  system 
arehiteeture — even  how  well  government  and  eontraetor  teams  interaet — make  it 
unrealistie  to  preseribe  a  one-size-fits-all  approaeh  to  eost  and  sehedule  eontrols 
(Johnson,  2006). 

Instead,  the  EVMS  Guidelines  provide  the  basis  for  determining  whether 
eontraetors'  EVM  systems  meet  standards.  These  broad  guidelines  serve  a  two-fold 
purpose  (Seott,  2005).  Eirst,  they  allow  for  eommon  sense  applieations  (read:  flexibility) 
on  both  sides  of  the  table — government  and  eontraetor.  Seeond,  their  eomprehensive 
nature  reassures  the  government  that  with  eaeh  report  it  reeeives  reasonably  reliable 
performanee  data. 

That  first  purpose  leads  to  the  eonelusion  that  eommon  sense  should  rule 
the  design,  employment,  and  subsequent  iterations  of  a  program’s  EVMS.  Elnfortunately, 
many  times  government  standard  operating  proeedures  and  eommon  sense  have  not 
positively  eorrelated.  More  often  than  not,  standard  operating  proeedures  meet  the  letter 
of  the  guidelines,  but  not  their  intent.  Eaeking  support  for  intent,  the  resultant 
ineongruenee  almost  always  eventually  fails  to  support  management's  needs,  and  the 
EVMS  inevitably  fails  as  a  management  eontrol  system  (Seott,  2005). 

The  seeond  purpose  depends  heavily  on  one  assumption  in  partieular:  the 
eontraetor  possesses  effeetive  internal  eontrols  (Seott,  2005).  The  presenee  of  these 
eontrols  makes  reliable  reporting  possible.  Without  that  foundation,  even  the  most 
stringent  proeess  attempting  to  follow  ANSI/EIA-748-A-1998  fails  proper 
implementation  due  to  uneertainty  surrounding  the  eost  and  sehedule  figures  produeed  by 
the  eontraetor. 

EVM  systems  that  eomply  with  the  intent  and  nature  of  the  guidelines 
faeilitate  projeet  work  seoped  in  its  entirety,  to  inelude  detailed  planning.  Properly 


5  Eleming  and  Koppelman  assert  that  industry  leaders  pereeived  the  previous  DoD- 
driven  standards  as  non-user-friendly  and  ineompatible  with  the  needs  of  private  industry. 
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implemented  EVM  systems  also  facilitate  full  integration  of  cost,  schedule,  and  project 
performance  objectives  into  a  performance  measurement  baseline  against  which  actuals 
(work,  cost)  can  be  measured.  An  effective  EVM  system  tailors  itself  to  a  given  program 
based  on  a  foundational  baseline  that  fosters  full  and  /  or  appropriate  control. 

Erom  a  reporting  standpoint,  an  effective  EVMS  uses  and  provides 
information  that  utilizes  the  defense  industry  standard  Work  Breakdown  Structure 
(WBS),  which  delineates  product  work  packages  as  well  as  organizational  responsibility. 
Within  each  WBS,  quantifiable  measurements  of  metrics — to  include  SV,  CV,  SPI,  CPI, 
etc. — should  generate  at  the  lowest  appropriate  organizational  levels  where  the  actual 
work  occurs.  Prom  those  fundamental  organizational  levels  up  through  the  highest  levels 
of  management,  the  EVMS  should  reflect  strong  discipline  in  reporting.  Otherwise, 
management  “dashboards”,  or  high-level  reporting  to  upper  management  used  for 
decision-making,  would  prove  ineffective. 

Prom  a  management  utilization  standpoint,  an  effective  EVMS  provides  a 
virtually  on-demand  analysis  of  significant  variances  (e.g.,  SV,  CV)  along  with  narrations 
of  forecasted  impacts.  An  effective  EVMS  becomes  a  key  enabler  of  management 
actions  that  may  mitigate  risk,  manage  cost,  and  manage  schedule.  Por  example,  the 
development  of  iterative  estimates  of  final  contract  costs,  beginning  with  the  initial  BAC 
and  ending  with  the  last  Eatest  Revised  Estimate  (ERE),  rates  as  both  management 
control  and  risk  moderator.  Effective  EVM  systems  impart  upon  a  program  at  least  a 
modicum  of  visibility  into  subcontractor  performance,  performance  that  directly  affects 
the  prime  contractor  (Scott,  2005). 

Having  seen  the  benefits  of  proper  guideline  adherence,  and  how  these 
yet-to-be-defined  guidelines  evaluate  an  EVMS,  the  next  section  contains  an  overview  of 
the  ANSI  EVMS  guidelines. 


c.  The  ANSI/EIA-748-A-1998  EVMS  Criteria 

Although  ANSI’s  EVMS  Criteria  may  have  changed  in  name  and  wording 
over  recent  years,  their  intent  has  remained  largely  unchanged  since  their  inception. 
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“Each  criterion  addresses  a  major  principle  neeessary  for  effeetive  management  of  large, 
flexibly  prieed  defense  projeets...eriteria  are  often  described  as  common-sense 
management  praetiees  that  any  well-managed  defense  eontractor  would  use” 
(Christensen,  1998).  Table  1  provides  a  eoneise  list  of  the  32  criteria. 
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Criteria 


Group  1:  Organization 

1 

Define  authorized  work  (WBS  elements) 

2 

Identify  organizational  responsibilities 

3 

Integrate  the  system 

4 

Identify  overhead  management 

5 

Provide  for  performanee  measurement 

Group  2:  Planning,  Scheduling  and  Budgeting 

6 

Sehedule  the  work 

7 

Identify  produets,  milestones  and  indieators 

8 

Plan  the  Performanee  Measurement  Baseline  (PMB) 

9 

Establish  budgets  for  work 

10 

Identify  work  paekages 

11 

Summarize  work  paekage  budgets  to  eontrol  aeeounts 

12 

Identify  and  eontrol  level  of  effort 

13 

Establish  overhead  budgets 

14 

Identify  management  reserves  and  undistributed  budget. 

15 

Summarize  budgets  to  target  eost 

Group  3:  Accounting 

16 

Reeord  direet  eosts 

17 

Summarize  direet  eost  to  the  WBS 

18 

Summarize  direet  eost  to  the  organization 

19 

Reeord  indireet  eosts 

20 

Identify  unit/lot  eosts 

21 

Reeord  material  eosts 

Group  4:  Analysis 

22 

Identify  sehedule  and  eost  varianees 

23 

Analyze  sehedule  and  eost  varianees 

24 

Analyze  indireet  eosts 

25 

Summarize  data  elements  and  varianees  for  reporting 

26 

Implement  managerial  aetions 

27 

Develop  revised  estimates  of  eost  at  eompletion 

Group  5:  Revisions 

28 

Ineorporate  ehanges  into  plans,  budgets  and  sehedules 

29 

Reeoneile  budgets  ehanges 

30 

Control  retroaetive  ehanges 

31 

Control  revisions  to  the  program  budget 

32 

Doeument  ehanges  to  the  PMB 

Table  1 .  ANSI/EIA-748-A- 1 998  EVMS  Criteria 
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As  presented  within  Table  1,  ANSI/EIA-748  organizes  the  eriteria  into  the 
following  five  areas  based  on  major  project  management  activities: 

•  Organization 

•  Planning,  Scheduling  and  Budgeting 

•  Accounting 

•  Analysis 

•  Revisions 

The  Organization  area  (five  criteria  total)  covers  the  definition  of 
authorized  work  within  a  program.  It  also  tasks  program  planners  to  ensure  some 
effective  delineation  of  organizational  structure  and  their  respective  responsibilities. 
Finally,  this  area  stipulates  some  integration  of  the  program’s  work  with  the 
organizational  structure  that  enables  effective  and  meaningful  measurements  of  cost  and 
schedule  performance. 

The  second  area.  Planning,  Scheduling  and  Budgeting  (ten  criteria  total), 
contains  information  regarding  proper  planning,  scheduling  and  budgeting  of  authorized 
work  so  information  gleaned  from  the  system  remains  meaningful.  Specifically,  this  area 
supports  and  explains  the  ideas  of  task  interdependency  awareness;  milestones,  delivery 
criteria,  and  other  measures  of  progress;  and  benefits  of  stable  and  measurable  units.  It 
also  references  some  areas  of  customer  interaction. 

The  Accounting  area  (six  criteria  total)  includes  a  discussion  of 
maintaining  accounting  discipline  so  information  remains  comparable  from  reporting 
period  to  reporting  period.  Not  surprisingly,  this  area  also  discusses  direct  costs,  indirect 
costs,  and  unit  costs  as  they  pertain  to  a  formal  EVMS.  Additionally,  this  area  discusses 
the  integration  of  a  material  accounting  system  with  the  planned  EVMS. 

The  next  area.  Analysis  (six  criteria  total),  suggests  the  frequency  of 
submitting  EVMS  reports  (at  least  monthly)  and  what  basic  data  to  include  (ACWP, 
BCWS,  BCWP,  SV,  CV).  It  also  presents  customer  reporting  requirements  and  pre- 
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requisites  for  delivery  of  a  meaningful  management  eontrol  produet.  This  area  also 
examines  implementing  ehanges  based  on  important  identified  varianees. 

Lastly,  Revisions  (five  eriteria  total)  diseusses  how  EVM  praetitioners 
should  ineorporate  ehanges  to  reports,  thereby  enabling  timely  and  effeetive  ehanges  to 
an  affeeted  program.  This  area  also  explains  the  differenee  between  appropriate  ehanges 
(e.g.,  eorreetion  of  errors)  and  inappropriate  ehanges  (e.g.,  hiding  flawed  information). 
Revisions  sets  forth  that  praetitioners  should  always  doeument  ehanges. 

With  the  eoneept  of  EVM  explained,  the  next  seetion  eontains  an  introduetion  to 
the  E-22  program. 

2,  The  F-22  Program 

In  the  summer  of  2002,  E-22  System  Program  Direetor  (SPD)  Brigadier  General 
William  J.  labour  eonfirmed  what  other  program  offieials  had  eautiously  hinted  at  for 
several  months:  the  P-22  would  miss  its  seheduled  start  date  for  the  program's  Dedieated 
Initial  Operational  Test  and  Evaluation  (DIOT&E)  (Chapman,  2002).  With  internal 
pressure  from  Air  Combat  Command  already  mounting  and  external  pressure  in  the  form 
of  eongressional  involvement  imminent,  the  timing  eould  not  have  been  worse  for  the 
foreeasted  six  month  sehedule  slip.  Senior  DoD  leaders  demanded  to  know  how  sueh  a 
monumental  program  failure  eould  oeeur  without  warning  and  talked  of  ominous 
eonsequenees  should  another  slip  transpire. 

Regardless  of  these  pressures  to  maintain  sehedule,  a  faet-fmding  group  known  as 
the  Red  Team  (assembled  by  eoneerned  program  proponents)  arrived  at  a  threatening 
eonelusion:  the  program  would  slip  again.  The  situation  reaehed  eritieal  mass  during  the 
elose  of  2002,  when  Seeretary  of  the  Air  Poree,  Dr.  James  G.  Roehe,  reassigned  Generals 
labour  and  Shaekelford  in  favor  of  “new  leadership... neeessary  to  aehieve  the  Air 
Poree’s  objeetives”  (Air  Poree  Print  News,  2002).  Internally,  Air  Poree  leadership 
wondered  how  this  eould  happen,  sinee  the  program  performanee  measurement  praetieed 
by  the  P-22  program  (i.e..  Earned  Value  Management)  should  eateh  impending  eost  and 
sehedule  problems  early  in  the  proeess. 
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In  fact,  three  years  earlier  the  Government  Aeeountability  Office  (GAO)  found 
that  the  F-22’s  prime  contractor,  Lockheed  Martin,  retained  reports  exhibiting  a 
downward  trend  with  respect  to  its  accomplishment  of  planned  work.6  Specifically, 
software  development  (i.e.,  avionics’  Operational  Flight  Program  (OFP))  for  the  F-22  fell 
behind  to  such  an  extent  that  a  rebaselining  of  schedule  oceurred  on  August,  1998. 
Aecording  to  the  GAO  (1999),  the  eauses  included;  avionics  development  falling  behind 
schedule,  unrealistie  avionics  schedule  goals,  and  the  eritieal  nature  of  avionics  with 
respect  to  the  weapon  system. 

This  schedule  slip  for  one  of  DoD’s  largest  weapons  system  acquisitions  joined  a 
long  list  of  timeline  adjustments  to  the  right.  Numerous  program  slips  oecurred  over  the 
F-22’s  twenty-plus  years  of  development.  Figure  1  shows  F-22  program  milestones 
eompared  to  other  legaey  aircraft  development  efforts.  The  cumulative  effect  of  all  F-22 
schedules  slips  resulted  in  the  program  taking  “76  pereent  longer  than  estimated  to 
achieve  first  flight  and  57  percent  longer  to  reach  first  production  [and]  19  percent  longer 
to  reach  Initial  Operational  Capability  (IOC)”  (Younossi,  Stem,  Lorell,  &  Fussier,  2005). 
One  driving  factor  is  persistently  manifested  behind  each  and  every  scheduling  failure: 
avionics  development. 


6  As  of  June  1998,  LMT  estimated  planned  work  not  completed  at  $1 15M. 
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Figure  1 .  F/A-22  Schedule  Slippage  Is  Higher  Than  the  Historical  Average 

(From:  Younossi  et  ah,  2005) 


Officially  designated  today  as  the  F-22A,  the  program  (formerly  known  as  the 
Advanced  Tactical  Fighter,  F-22,  and  F/A-22)  has  undergone  dramatic  mission 
requirement  changes  since  its  inception  in  the  late  1980’s.  Originally,  US  leadership 
envisioned  the  F-22  as  an  answer  to  the  Soviets’  Su-27  and  MiG-29  aircraft  that 
threatened  to  technologically  usurp  the  global  dominance  of  the  Boeing  F-15  fleet. 
However,  in  the  eyes  of  budget  hawks  the  fall  of  the  Soviet  Union  (and  subsequent 
termination  of  the  Cold  War)  changed  the  requirement  for  a  next  generation  air-to-air 
platform.  Many  questioned  the  need  to  move  away  from  the  historical  dominance  of  the 
current  F-15  fleet  in  a  world  lacking  adversarial  nation-states  with  upgraded,  current 
generation  tighter  aircraft. 

This  new  global  reality  forced  the  Air  Force  to  move  away  from  a  strict  air-to-air 
role  and  instead  introduce  air-to-ground  requirements  to  the  F-22  program.  While  the 
advanced  air-to-air  capabilities  of  stealth,  supercruise,  and  integrated  avionics  remained 
the  foundation  of  the  revolutionary  tighter,  the  program  solidified  plans  that  would 
incrementally  add  additional  capabilities  such  as  delivery  of  Joint  Direct  Attack 
Munitions  and  enhanced  air-to-ground  radar  to  the  jet.  As  a  result,  the  Air  Force  directed 

the  F-22  System  Program  Office  (SPO)  to  initiate  a  new  Modernization  program,  with  its 
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main  objective  being  the  development  of  in-line  and  post-production  upgrades  to  the 
fighter.  Though  joint  SPO  and  Contractor  Team  plans  to  integrate  a  majority  of  these 
new  requirements  looked  to  post-EMD,  several  key  requirement  changes  required 
introduction  and  implementation  prior  to  completion  of  both  EMD  and  approval  to 
proceed  with  Eull  Rate  Production  (Younossi  et  ah,  2005). 

While  many  defense  experts  typically  point  to  the  introduction  of  these  new 
requirements  as  a  key  driver  behind  the  schedule  slips  and  accompanying  cost  over-runs, 
others  note  that  the  P-22  experienced  significant  cost  and  schedule  variances  prior  to 
introduction  of  these  additional  requirements.  Por  example,  according  to  a  2006  study 
by  Younossi  et  ah,  “[P-22]  cost  growth  was  mainly  the  result  of  design  challenges  in  the 
airframe  (arising  from  stealth  requirements),  the  integrated  avionics  suite,  and  the  new 
propulsion  system.”  Pigure  2  highlights  this  statement  by  presenting  the  cost  growth  of 
the  P-22  by  major  system  from  1995  through  2002. 


Figure  2.  F-22  Cost  Growth  Trends  for  Major  Systems 

(From:  Younossi  et  ah,  2005) 


Pooking  at  percentage  of  cost  growth  alone  rates  the  F-22’s  integrated  avionics 


suite  as  one  of  the  areas  of  greatest  concern.  The  highly  complex  avionics  subsystem, 
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touted  as  one  of  the  three  key  teehnologieal  advaneements  contributing  to  the  dominating 
existence  of  the  F-22,  certainly  explains  that  cost  growth.7  After  all,  advancements  such 
as  those  fielded  in  the  avionics  suite  do  not  come  without  an  appropriate  price  tag.  In 
fact,  government  experts  calculated  this  subsystem  consumed  one-third  of  the  F-22 
program  budget — “more  than  any  other  subsystem,  including  the  airframe”  (Younossi  et 
ah,  2005).  This  still  leaves  the  question  of  “What  happened  that  caused  such  price 
growth?” 

One  answer  may  lie  in  a  comparison  of  the  historical  development  of  avionics 
systems  and  the  F-22’s  systems.  While  legacy  aircraft  avionics  followed  a  federated 
construct  where  each  avionics  subsystem  (e.g..  Communications  /  Navigation  / 
Identification,  Electronic  Warfare,  Radar,  etc.)  provided  information  to  the  pilot 
independently  from  other  subsystems,  the  F-22  uses  a  central  core  processor  to  fuse  this 
information  from  the  various  sensors  and  other  components  to  present  an  integrated 
picture  to  the  pilot.  This  requires  extremely  large  numbers  of  instructions  per  second — 
millions  for  data  processing  and  billions  for  signal  processing — creating  extensive 
demands  on  aircraft  computing  systems  that  resulted  in  significant  system  lock-ups 
during  developmental  testing  (GAO,  2004). 

In  addition  to  the  extreme  demands  on  avionics  hardware,  the  F-22  requires 
software  complexity  at  an  unprecedented  level  to  manage  the  data  flowing  through  said 
hardware.  Software  designers  answered  that  need  with  the  F-22  Operational  Flight 
Program.  But  once  again,  increasing  complexity  and  requisite  robustness  became  another 
driver  that  increased  avionics  cost.  Between  October  1993  and  April  2000  the  F-22 
Software  Lines  of  Code  (SLOC)  grew  approximately  34  percent  (Younossi  et  ah,  2005). 
According  to  the  GAO,  this  SLOC  growth,  largely  driven  by  requirements  and  design 
changes,  resulted  in  delayed  software  deliveries,  impacting  program  cost  and  schedule. 


7  The  other  two  L-22  “first  look,  first  shot,  first  kill”  capabilities  identified  as  “firsts” 
in  US  military  aircraft  are  supercruise  and  stealth.  The  powerful  L-119  engines  and 
airframe  design  provide  supercruise  capability,  enabling  the  L-22  to  cruise  at  supersonic 
speeds  without  the  use  of  afterburners.  Although  other  aircraft  have  fielded  stealth 
technology,  the  L-22  is  noted  as  being  the  US  first  all-weather,  “24-7-365”  stealth  tactical 
fighter. 
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and  “accounted  for  37  percent  of  the  eritical  problems  reports  leading  to  avionics 
shutdowns  in  the  [F-22].”  (2004) 

Conventional  wisdom  supports  the  idea  that  the  Air  Foree — not  to  mention  other 
serviees — would  like  to  avoid  similar  situations  in  the  future.  Aiding  that  desire  beeame 
the  ultimate  driver  behind  this  study.  The  next  seetion,  Purpose  and  Significance  of 
Study,  further  explores  the  motivating  forees  behind  this  researeh. 


B,  PURPOSE  AND  SIGNIFICANCE  OF  STUDY 

Within  the  eontext  of  this  study,  two  goals  remained  paramount  and  thus  defined 
the  projeet’s  purpose.  The  first  goal:  determine  if  eurrent  EVM  implementation  within 
the  F-22’s  Modernization  program  will  likely  prevent  an  abrupt  sehedule  slip  and  eertain- 
to-follow  eost  overrun — similar  to  what  oecurred  during  EMD.  The  seeond  goal:  make 
meaningful  recommendations,  where  appropriate,  with  the  objeetive  of  strengthening  the 
eurrent  EVM  system  implemented  to  measure  the  E-22  program’s  software  development 
performance.  Again,  these  goals  delineate  the  study’s  purpose. 

The  signifieanee  of  the  study  relied  upon  the  expectation  of  ever-increasing 
weapon  system  eomplexity,  espeeially  within  the  realm  of  software  integration.  Eor 
example,  upon  completion  of  a  Program  Offiee  Estimate  (POE)  on  Mareh  of  2003,  the 
government  predicted  E-22  EMD  eosts  to  exceed  Milestone  II  estimates  by  33  pereent. 
Granted,  a  program  spanning  an  inordinately  large  number  of  years  should  expeet  to  see 
mission  requirement  and  teehnology  ehanges  that  drive  sehedule  and  eost  impaets, 
however,  the  E-22  aetually  fared  worse  than  the  average  of  other  similar  development 
efforts,  with  a  sehedule  growth  factor  of  1.33  versus  a  1.22  average  (see  Eigure  3  below). 
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Figure  3.  F/A-22  Cost  Growth  Is  Fligher  Than  the  Historieal  Average 

(From;  Younossi  et  ah,  2005,  p.  10) 


As  evideneed  within  the  first  section,  Background,  avionics  development  rated  as 
a  significant  cost  driver.  When  considered  concurrently  with  the  schedule  trials 
experienced  during  EMD,  these  facts  escalated  the  importance  of  discovering  whether  or 
not  the  EVMS  implemented-of-late  worked  to  minimize  the  schedule  and  cost  risks  of  an 
unprecedented  software  effort.  More  importantly,  given  the  increasing  cost  of  warfare- 
dominating  technology,  DoD  should  address  this  prior  to  reaching  comparable  phases  of 
development  within  upcoming  fiagship  acquisitions:  the  Joint  Strike  fighter  and  the 
future  Combat  System.  According  to  the  GAO,  these  two  programs  are  projected  to  cost 
DoD  over  $330B  —  more  than  five  times  the  total  cost  of  the  f-22  program  and  roughly 
80  percent  of  DoD’s  entire  FY07  Appropriation  Bill  (2006). 

Thus,  learning  from  mistakes  of  the  past  becomes  vital  within  DoD’s  resource- 
constrained  environment.  Seeking  significant  answers  lies  in  asking  pointed  questions, 
first,  if  the  F-22  program  office  practiced  and  used  EVM,  how  did  these  problems  appear 
to  catch  everyone  (but  program  detractors,  of  course)  by  surprise?  Easily,  why  weren’t 
the  abrupt  cost  overruns  and  schedule  failures,  that  cost  an  SPD  his  job,  caught  earlier? 
The  next  section.  Research  Questions,  further  explores  and  refines  these  questions  by 
putting  them  into  the  context  of  this  study. 
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C.  RESEARCH  QUESTIONS 

Finding  answers  to  a  problem  entails  first  defining  the  problem.  Three  direet 
questions  defined  the  problem-at-hand  by  asking; 

•  How  closely  did  the  F-22  Spiral  2  implementation  of  EVM  follow  the 
criteria  outlined  in  ANSI/EIA-748  Earned  Value  Management  System? 

•  To  what  degree  did  the  E-22  Spiral  2  implementation  of  EVM  fulfill  its 
role  as  a  management  control  system  for  avionics  software  development? 

•  To  what  extent  did  the  E-22  Spiral  2  program  management  (Government 
and  Contractor)  use  EVM  products  to  manage  avionics  development 
efforts? 

Answering  these  questions  assessed  the  E-22  program’s  use  of  EVM  in  managing 
avionics  software  development  from  a  current  (Spiral  2)  perspective.  The  section-to- 
follow  describes  the  approaches  taken  to  answer  these  questions  and,  more  importantly, 
explains  why  this  subject  matter  was  chosen  for  academic  research. 


D,  METHODOLOGY 

This  section  begins  with  a  brief  explanation  of  why  the  E-22  Spiral  2  program  was 
chosen  as  a  representative  case  study  of  the  interaction  between  DoD  software 
development  and  the  EVMS  that  measures  its  progress.  As  established  in  the  previous 
section,  three  questions  essentially  frame  the  research  problem;  this  section  concludes  by 
matching  these  same  questions  with  their  primary  answering  method.  8 

Why  choose  the  E-22  Spiral  2  program  as  a  case  study?  The  first  and  foremost 
reason;  access.  Utilizing  the  professional  relationships  and  contacts  resulting  from  prior 
experience  at  the  E-22  SPO  greatly  enhanced  the  probability  of  both  a  successful  research 
effort  and  the  delivery  of  useful  recommendations.  Access  eased  not  only  asking  the 


8  The  primary  distinction  was  made  since,  in  reality,  each  answer  more  or  less  utilized 
a  mix  of  methods,  with  the  primary  method  answering  the  majority  of  issues  relating  to 
its  respective  question. 
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questions  and  collecting  answers;  it  facilitated  the  project  with  the  knowledge  of  whom  to 
ask. 

Along  the  same  lines,  the  first-hand  knowledge  and  direct  observations  gained 
from  prior  experience  at  the  F-22  SPO  created  a  sense  of  familiarity  within  the  context  of 
a  research  project  seeking  an  unknown  answer.  This  fomented  a  synergistic  effect 
between  knowledge  gained  while  attending  the  Naval  Postgraduate  School  (NPS)  and  the 
knowledge  already  possessed  of  EVM  standard  operating  procedures  at  the  F-22 
program.  Consequently,  this  allowed  the  research  to  move  beyond  basic  concepts  and 
into  the  world  of  practical  application,  even  prior  to  the  information  gathering  phase. 

Another  reason  that  supported  studying  the  F-22  program  was  the  program’s 
pending  universality.  Simply  stated,  pending  universality  means  that  the  F-22  program’s 
software  development  exhibits  trends9  that  experts  believe  will  become  commonplace  in 
future  DoD  acquisition  efforts.  As  such,  having  studied  the  interactions  between 
unprecedented  software  development  and  FVM,  resulting  universal  concepts  (i.e., 
concepts  applying  to  all  programs,  regardless  of  function)  from  the  study  should  at  least 
partially  transfer  to  upcoming  programs. 

The  final  reason  behind  choosing  the  F-22  program  proved  less  complicated:  the 
simple  desire  to  help.  The  lack  of  clear-cut  answers  to  the  research  problems,  even  with 
previous  experience  with  the  program,  fostered  unease  with  respect  to  FVM 
implementation  within  software  development  programs.  Only  after  the  attainment  of  new 
academia-based  knowledge  while  attending  NPS  (e.g..  Acquisition,  Cost  Fstimation,  and 
Research  courses)  was  it  felt  that  a  helpful  answer  could  become  perceptible. 

In  summary,  the  F-22  Spiral  2  program  was  chosen  as  the  program-of-study  due 
to  its  relative  ease  of  access,  familiarity,  pending  universality,  and  a  desire  to  help  the 
program.  The  following  paragraphs  concern  the  applications  to  the  research  problems 
introduced  in  the  earlier  Research  Questions  section. 


9  Specifically,  the  Joint  Strike  Fighter  and  Future  Combat  System  both  exhibit  similar 
degrees  of  software  complexity  resulting  from  the  weapon  systems’  increased 
dependency  on  software  and  integrated  constructs. 
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Answering  “How  closely  did  the  F-22  Spiral  2  implementation  of  EVM  follow 
the  criteria  outlined  in  ANSI/EIA-748-A-1998  Earned  Value  Management  System?” 
relied  mainly  on  a  careful  assessment  of  how  EVM  implementation  for  Spiral  2 
supported  the  32  ANSI/EIA  criteria.  The  text  Earned  Value  Project  Management  by 
Eleming  and  Koppelman  provided  a  majority  of  the  assistance  through  its  detailed 
description  of  each  of  the  criteria.  Those  descriptions  were  aligned  with  procedures  in 
place  within  the  E-22  software  program,  in  turn  identified  by  interviews,  portions  of  the 
questionnaire,  and  collected  management  documents.  The  degree  of  alignment  answered 
the  question  for  each  of  the  criteria. 

“To  what  degree  did  the  E-22  Spiral  2  implementation  of  EVM  fulfill  its  role  as  a 
management  control  system  for  avionics  software  development?”  was  answered 
primarily  by  data  collection.  Specifically,  Cost  Performance  Reports  (CPRs)  directly 
related  to  Spiral  2  efforts  were  distilled  into  raw  cost  and  schedule  measurements.  A 
complete  EVM  analysis  was  accomplished  and  compared  to  outputs  from  SPO  and  GAO 
reports,  in  addition  to  testimony  from  individuals  involved.  The  comparison,  in  essence, 
answered  the  question. 

Easily,  the  EVM  Questionnaire  answered  the  question  “To  what  extent  did  E-22 
program  management  (Government  and  Contractor)  use  EVM  products  to  manage 
avionics  development  efforts?”  The  questionnaire  aimed  to  assess  the  perceived 
usefulness  of  EVM  within  a  software  development  context.  If  a  given  respondent 
answered  positively  to  that  question,  they  were  asked  to  rate  the  practical  value  of  EVM 
with  respect  to  their  program  management  duties.  Interviews  supplemented  the  data  from 
the  questionnaire,  figuratively  filling  in  the  questionnaire’s  information  gaps  uncovered 
throughout  the  course  of  the  research  project. 

In  summary,  accomplishing  this  project  involved  conducting  interviews  with 

subject  matter  experts,  both  at  the  E-22  SPO  and  Eockheed  Martin,  to  assess  their 

thoughts  on  exactly  how  well  the  program  followed  the  ANSI/EIA-748-A-1998 

guidelines.  In  addition,  results  from  the  EVM  Questionnaire  were  reported,  along  with 

deduced  conclusions.  This  questionnaire  surveyed  occupational  specialties  involved  with 

software  development  efforts,  and  summarizes  opinions  and  knowledge  related  to  EVM 
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and  ANSI/EIA-748-A-1998.  Finally,  data  and  report  collection  assessed  the  information 
provided  by,  among  other  sources,  the  avionics  development’s  EVMS. 


E,  FRAMEWORK 

Chapter  I,  Introduction,  served  three  main  purposes.  First,  the  chapter  provided 
context  by  discussing  the  history  of  avionics  development  within  the  larger  F-22 
development  effort  (see  the  Background  section).  Second,  it  revealed  the  paper’s  purpose 
and  why  this  study  may  prove  significant  to  future  DoD  efforts.  Finally,  it  established  an 
academic  framework  by  identifying  the  problems  this  paper  seeks  to  answer  and  how  it 
answered  them  (see  the  Research  Questions  and  Methodology  sections). 

Chapter  II,  Literature  Review,  provides  an  informed  foundation  by  examining 
current  bodies  of  work  that  discussed  applicable  topics.  Since  research  efforts  focused  on 
both  software  development  and  EVM,  relevant  information  included  references  to 
suitable  texts  containing  foundational  thinking  associated  with  these  two  topics — to 
include  a  basic  primer  for  EVM.  Additionally,  contemporary  ideas  regarding  the 
interaction  between  EVM  and  software  development  were  explored. 

Chapter  III,  F-22  Implementation  of  ANSI/EIA  EVMS  Criteria,  highlights  the  32 
ANSI  criteria.  Using  those  criteria — as  explained  in  the  Earned  Value  Project 
Management  text — an  assessment  was  made  of  EVM  implementation  within  F-22 
software  development.  Each  criterion  was  analyzed  separately  and  a  conclusion  was 
reached  regarding  the  degree  of  alignment  between  implementation  and  its  intended 
purpose. 

Chapter  IV,  F-22  EVMS  Environment,  examines  the  recent  and  current 
environment  of  EVM  as  it  pertains  to  F-22  software  development.  This  chapter  seeks  to 
consolidate  questionnaire  responses,  interviews,  observations,  data  collected,  and  the 
authors’  experiences  regarding  how  the  F-22  program  applies  EVM  to  software 
development.  Specifically,  the  chapter  aims  to  provide  insight  into  how  the  current 
avionics  contract  environment,  avionics  suppliers,  and  recent  avionics  programs 
themselves  may  or  may  not  contribute  to  difficulties  in  EVMS  implementation. 
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Chapter  V,  Conclusion,  presents  a  eondensed  synopsis  of  this  researeh  project’s 
outcome,  includes  a  brief  discussion  on  limitations  with  respect  to  the  research  project, 
and  makes  final  recommendations  to  the  EVMS  where  necessary.  According  to  this 
paper’s  research,  these  recommendations  (if  required)  should  serve  to  strengthen  the  F-22 
avionics  program’s  EVM  system. 

F.  SUMMARY 

This  concludes  Chapter  I,  Introduction.  The  next  chapter.  Literature  Review, 
introduces  the  reader  to  existing  academic  works  associated  with  Software  Development, 
and  EVM,  and  the  interaction  between  the  two.  Chapter  II  also  includes  further 
background  on  E-22  avionics  software  development. 
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II.  LITERATURE  REVIEW 


A.  PREFACE 

This  chapter  strives  to  take  the  research  topic,  An  Analysis  of  Earned  Value 
Management  Implementation  in  the  F-22  System  Program  Office ’s  Software 
Development,  and  provide  the  reader  with  an  informed  and  expert-based  framework  using 
a  diverse  eollection  of  reports,  papers,  data,  and  experience.  Providing  the  framework 
itself  entailed  examining  existing  bodies  of  work  that  discuss  the  following: 

•  Unique  Aspects  of  Software  Development 

•  Lessons  Learned  Regarding  the  F-22  Program  and  Software  Development 

•  Current  F-22  Software  Development  Strategy — Spiral  Modernization 

A  eompleted  analysis  of  these  three  areas  will  not  only  set  the  stage  for  the  remainder  of 
the  paper,  it  will  also  enable  the  reader  to  begin  framing  desired  scenarios  against  the 
backdrop  of  these  real-world  challenges  and  proeesses. 

B,  UNIQUE  ASPECTS  OF  SOFTWARE  DEVELOPMENT 

1.  Software  Development  Challenges 

Today  the  F-22  exists  as  one  platform  amid  an  exploding  population  of  DoD 
equipment  now  relying  heavily  on  software  to  perform  their  respective  missions.  Even 
so,  the  F-22  remains  a  precursor  to  the  major  defense  weapons  systems  of  tomorrow 
through  its  use  of  complex,  embedded  software.  Consider  the  following:  aceording  to  a 
Defense  Science  Board’s  Task  Force  on  Defense  Software  report  (2000),  military  aircraft 
dependency  on  software  inereased  from  approximately  10  pereent  funetionality  on  the  F- 
4  to  80  pereent  functionality  on  the  F-22 — equivalent  to  a  2  pereent  per  year  increase 
(1960-1995).  Simply  stated,  software  has  become  ubiquitous  within  DoD  aequisitions, 
and  today’s  high  tech  machines  of  war,  in  fact,  depend  on  it  at  unpreeedented  levels. 

For  example,  even  DoD’s  historieally  “dumb”  weapons — items  such  as  air-to- 
ground  bombs  and  artillery  rounds — rate  modernization  funding  to  equip  them  with 
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advanced  systems  that  boost  functionality,  precision,  and  lethality.  However,  equipping 
sueh  munitions  with  GPS  systems  and  autonomous  guidanee  technologies  predictably 
increases  their  own  reliance  on  software  to  perform  their  functions.^*’  As  a  result,  a  given 
inerease  in  performance  eauses  some  commensurate  increase  in  complexity  and  risk  (i.e., 
the  inherent  trade-offs  between  performance,  sehedule,  and  cost). 

A  1999  study  performed  by  the  Standish  Group  (an  organization  that  studies 
information  technology  investments)  brought  attention  to  this  suspected  trend.  The 
study  found  31  percent  of  commereial,  DoD,  and  combined  commereial-DoD  software 
development  efforts  resulted  in  eancellation.  In  addition,  the  study  reported  the  following 
software  development  statistics  (GAO,  2004);  n 

•  Cost  overruns  of  1 89  pereent 

•  Schedule  delays  of  222  percent 

•  Delivery  of  61  pereent  of  originally  specified  features  or  functions 

In  an  attempt  to  find  some  root  cause  of  these  dismaying  statistics,  the  Defense 
Science  Board,  once  again,  investigated  the  area  of  software  development.  The  Board 
found  that  software-intensive  “programs  lacked  a  well  thought-out,  disciplined  program 
management  and/or  software  development  proeesses.”  The  findings  went  on  to  state  that 
“meaningful  cost,  schedule,  and  requirements  baselines  were  lacking,  making  it  virtually 
impossible  to  track  progress  against  them”  (2000). 

Exploring  this  concept  further  involves  analyzing  the  key  differences  between 
hardware  and  software  development.  For  instance,  when  compared  to  hardware,  software 
tends  to  propagate  ehange  effects  at  a  higher  rate.  Furthermore,  software  exists  more  in 
the  intangible  realm  of  data  and  logic,  versus  physical  components.  Finally,  software  has 


10  Examples  are  the  US  Air  Foree  Small  Diameter  Bomb  (SDB)  and  US  Army 
Exealibur  weapons.  The  SDB  program  is  a  250-lb  class  guided  munition  eurrently  under 
development  at  the  USAF  Air  Armament  Center  (Picatinny  Arsenal  News  Release, 
2005).  The  US  Army’s  Exealibur  program  is  a  howitzer-fired  munition  that  uses  GPS  to 
guide  (in  flight)  to  within  10  meters  of  its  target  (Ruscetta,  2005). 

11  Percentages  were  based  on  comparison  with  initial  baseline. 
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limited  standardized  design  methods,  components  or  structure  when  matched  up  against 
hardware. 

These  differences  mean  that  a  typical  software  development  project 
underestimates  the  development  schedule  when  planners  employ  methods  used  to  predict 
non-software  development.  It  appears  the  very  nature  of  software  causes  development 
issues  that  translate  into  cost,  schedule,  and  performance  concerns.  Therefore,  while 
labeling  a  development  effort  as  “hardware”  certainly  does  not  grant  immunity  from 
these  challenges  and  critiques,  the  complexity  and  prevalence  of  embedded  software  in 
weapons  systems  heightens  the  probability  that  problems  do  occur. 

Software’s  nature  does  not  completely  differentiate  itself  from  hardware, 
however.  Software,  in  many  cases,  requires  full  integration  with  hardware.  As  such,  it 
must  share  at  least  some  fundamental  characteristics  with  hardware  to  facilitate  said 
integration.  The  Defense  Acquisition  University  (DAU)  stated  that  some  of  the  key 
similarities  between  hardware  and  software  include:  functional  decomposition; 

traceability  to  system  requirements;  accountability  by  task;  progress  monitoring;  and 
reliance  on  operating  principles  and  constraints  (2006). 

As  for  DoD  specific  studies  on  the  matter,  the  Acquisition,  Technology,  and 
Logistics  (AT&L)  Knowledge  Sharing  System  (AKSS)  summarized  the  typical  problems 
defense  acquisition  programs  have  encountered  over  the  years.  For  one,  the  dynamic  and 
rapidly  evolving  nature  of  software  development  makes  it  difficult  to  adhere  to  an 
original  baseline.  Program  managers  and  other  decision  makers  lack  basic  software 
knowledge,  which  only  aggravates  the  problem  of  baseline  adherence. 

Similarly,  end-product-users  typically  cannot  accurately  convey  requirements, 
and  promulgate  requirements  creep  12  throughout  the  development  phase  (once  again 
hampering  the  baseline).  This  problem  in  turn  leads  to  joint  software  and  hardware 
development  either  starting  or  becoming  uncoordinated,  either  directly  because  of  poor  or 


12  “A  tendency  for  product  or  project  requirements  to  increase  during  development 
beyond  those  originally  foreseen,  leading  to  features  that  weren't  originally  planned  and 
resulting  risk  to  product  quality  or  schedule”  (Johnson,  2005). 
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non-existent  software  development  metries  or  as  a  result  of  inadequate  software  testing 
programs. 

AKSS  provides  a  final  assertion  that  effeetively  ereates  a  foundational 
predieament  with  respeet  to  the  previously  listed  problems.  DoD  personnel  generally 
(with  few  exeeptions)  laek  fundamental  software  development  knowledge:  that  dearth  of 
knowledge  will  continue  into  the  foreseeable  future  until  DoD  can  effectively  compete 
with  the  private  industry  for  software  engineers. 

This  segment  contained  a  brief  discussion  of  software  development  challenges; 
the  next  topic  concerns  the  distinctive  software  development  lifecycle,  and  uses  this 
context  to  further  compare  and  contrast  software  with  hardware. 

2.  Software’s  Lifecycles 

Fundamental  characteristics  aside,  other  differences  pertinent  to  this  paper  exist 
between  hardware  and  software  development.  Professional  organizations  such  as  the 
Carnegie  Mellon  Software  Engineering  Institute  (SEI)  and  the  USAE  Software 
Technology  Support  Center  (STSC)  recognized  this  and  bestowed  a  unique  lifecycle 
upon  software  development  separate  from  hardware  development.  Although  different 
than  the  lifecycle  phases  typically  imparted  to  classic  hardware-intensive  efforts,  note  the 
similarities  and  parallels  between  the  phases  of  the  software  lifecycle  and  those  of 
hardware  (2005;  2003): 

•  Requirements  -  Through  interfacing  with  the  customer,  the  developer 
analyzes  operational  problems  or  needs  and  translates  them  into  functional 
requirements.  This  Systems  Engineering  process  results  in  lower-level, 
detailed  functional  requirements  traceable  to  higher-level  requirements. 
Contrast  to  Concept  Refinement  and  Technology  Development  within  the 
Acquisition  Eramework  (Eigure  4). 

•  Design  -  This  phase  involves  definition  of  the  software  structure.  It 
analyzes  specific  solutions  and  approaches  and  chooses  the  best 
alternative  based  on  cost,  schedule  and  performance  parameters.  Two 
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design  reviews  within  this  phase  typically  approve  the  Preliminary  Design 
(the  initial  software  architecture)  and  the  Detailed  Design  (functional 
modules  and  interfaces).  Contrast  to  Technology  Development  and  System 
Development  and  Demonstration  within  the  Acquisition  Framework 
(Figure  4). 

•  Implementation  (Development)  -  This  phase  involves  actual  coding  of 
software.  Coding  usually  entails  an  iterative  approach  consisting  of 
subsystem  (component)  unit  development  and  testing  prior  to  integration 
testing  within  the  main  software  build.  Results  from  that  testing  in  turn 
help  develop  yet  another  round  of  coding.  Contrast  to  System 
Development  and  Demonstration  within  the  Acquisition  Framework 
(Figure  4). 

•  Testing  -  This  phase  typically  involves  three  types  of  testing:  Unit 
Testing,  Integration  Testing,  and  Acceptance  Testing.  As  discussed 
above,  accomplishing  the  first  two  types  entails  many  cyclical  trials  prior 
to  proceeding  with  acceptance  testing,  which  verifies  performance  against 
requirements.  Contrast  to  System  Development  and  Demonstration  within 
the  Acquisition  Framework  (Figure  4). 

•  Deployment  -  Anticipate  this  phase  to  field  the  software  product  in  its 
intended  environment.  Also,  users  become  familiar  with  the  system  via 
training.  Once  complete,  it  finalizes  the  system  development  effort. 
Contrast  to  Production  and  Deployment  and  Operations  and  Support 
within  the  Acquisition  Framework  (Figure  4). 

•  Maintenance  -  Depending  on  the  need  for  enhancement,  fixes,  or 
modifications,  this  phase  ranges  in  scope  from  a  minimal  to  a  Herculean 
effort  even  larger  than  the  original  development.  This  phase  typically 
costs  far  more  than  the  original  development  effort.  Changes  in  software 
this  late  in  development  come  with  a  heftier  price  tag  than  the  same  effort 
undertaken  during  an  earlier  phase.  Contrast  to  Production  and 
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Deployment  and  Operations  and  Support  within  the  Acquisition 
Framework  (Figure  4). 
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Figure  4.  Comparing  Defense  Acquisition  Management  &  Software  Acquisition 

Framework 

(From;  AT&L  Knowledge  Sharing  System,  2006;  MN3301,  2006) 
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As  alluded  to  within  the  previous  section,  one  of  the  inherent  challenges  with 
software  development  efforts  lies  in  premature  migration  into  the  implementation  phase 
prior  to  sufficient  completion  of  the  requirements  and  design  phases.  In  an  assessment  of 
commercial  software  development  companies,  the  GAO  identified  best  practices  that 
included  the  need  for  management  to  protect  against  missing,  vague  or  changing 
requirements  that  negatively  impact  programs.  Using  commercial  industry  as  the 
standard,  GAO  identified  a  benchmark  of  setting  95  percent  of  requirements  by  the  end 
of  the  requirements  phase,  and  98  percent  by  the  end  of  the  design  phase  (GAO,  2004). 
For  the  military,  the  need  for  fully  defined  and  stable  requirements  presents  a  unique 
challenge.  In  its  report  to  the  Secretary  of  Defense  in  2006,  the  Defense  Acquisition 
Performance  Assessment  panel  stated  the  following: 

The  [DoD]  Acquisition  System  must  deal  with  external  instability,  a 
changing  security  environment  and  challenging  national  security  issues. 

The  Department  must  be  agile — to  an  unprecedented  degree — to  respond 
quickly  to  urgent  operational  needs  from  across  the  entire  spectrum  of 
potential  conflicts,  (p.  7) 

Balancing  this  need  for  flexibility  against  the  recognized  need  for  requirements  stability 
in  software  development  heightens  the  need  for  flexible,  disciplined  program  controls 
within  well-managed  programs. 

Defining  and  understanding  the  different  phases  of  the  software  lifecycle  is  only 
the  first  step  to  successfully  managing  a  software  program.  As  with  any  project  planning 
effort,  managers  must  identify  those  critical  factors  necessary  to  determine  a  software 
program’s  success  or  failure.  The  five  critical  factors  identified  by  the  Air  Force’s  STSC 
are:  Quality,  Cost,  Schedule,  Performance  and  Supportability.  For  each  of  these  factors, 
the  project  manager  must  develop  appropriate  plans,  criteria,  expectations,  measures  and 
controls  to  ensure  the  program  stays  on  course.  Since  its  inception,  EVMS  has  proven  a 
powerful  tool  for  measuring  and  controlling  the  factors  of  cost  and  schedule.  Although  it 
“requires  a  fully  defined  project  up  front  and  bottom-up  cost  estimates... it  can  provide 
accurate  and  reliable  indication  of  cost  performance  as  early  as  15  percent  into  the 
projecf’  (STSC,  2000). 
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This  section  defined  and  represented  the  difficulties  implicit  within  a  software 
development  program.  The  next  section  discusses  the  performance  measurement  of 
software  development  within  the  context  of  the  F-22  program. 

C.  LESSONS  LEARNED  REGARDING  THE  F-22  PROGRAM  AND 

SOFTWARE  DEVELOPMENT 

Current  research  pertaining  to  measuring  the  progress  of  software  development 
supports  the  assertion  that,  in  the  initial  stages  of  development,  software  efforts  track 
much  like  their  hardware  counterparts.  In  other  words,  it  is  relatively  easy  to  apply  EVM 
to  the  first  two  phases — Requirements  and  Design.  However,  this  assumes  that  the 
program  in  question  has  adequate  cost  and  schedule  controls. 

Two  recent  assessments  of  the  F-22  program  provided  recommended  changes 
regarding  the  interactions  between  its  software  development  and  its  EVMS.  Between  the 
assessments  exists  a  common  theme:  the  program  needs  better  cost  and  schedule 
controls.  The  first  report  was  written  by  the  Government  Accountability  Office  (GAO); 
the  RAND  Corporation  generated  the  second  report. 

In  its  2004  report,  DEFENSE  ACQUISITIONS:  Stronger  Management  Practices 
Are  Needed  to  Improve  DoD’s  Software-Intensive  Weapon  Acquisitions,  the  GAO 
recommended  DoD  require  its  software-intensive  development  contractors  to  first  collect 
and  regularly  report  metrics  related  to  software  cost,  schedule,  size,  requirements,  tests, 
defects  and  quality.  In  its  next  recommendation,  the  GAO  suggests  that  DoD,  in 
cooperative  effort  with  its  contractors,  develop  “an  earned  value  management  system  that 
reports  cost  and  schedule  information  at  a  level  of  work  that  provides  information 
specific  to  software  development.” 

More  recently,  in  2005  Younossi  et  al.  identified  in  Lessons  Learned  from  the 
F/A-22  and  F/A-I8E/F  Development  Programs  the  need  to  have  EVM  data  “monitor  and 
manage  program  costs  at  the  level  of  integrated  product  teams.”  However,  merely  stating 
these  controls  should  be  put  in  place  doesn’t  necessarily  equate  to  examination  and 
understanding  of  all  implications  related  to  the  complexities  of  applying  EVM.  This  is 
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especially  evident  when  considering  the  complex,  dynamic,  and  unique  characteristics  of 
software  development. 

Avionics  remains  a  critical  and  arguably  the  most-complicated  system  of  the  F- 
22.  It  heavily  impacts  both  cost  and  schedule,  and  has  done  so  over  a  significant  period 
of  the  program.  Both  program  and  contractor  officials  admitted  that  the  program  failed  to 
follow  their  stated  software  strategy:  to  collect  metrics  and  manage  to  those  metrics. 
This  failure  facilitated  the  loss  of  program  cost  and  schedule  control.  Further 
investigation  revealed  that  other  cost  and  schedule  pressures  within  the  F-22  program 
contributed  to  a  failure  of  desired  software  metrics.  These  pressures  kept  the  program 
from  providing  its  managers  the  necessary  metrics  for  sufficient  oversight  of  the  overall 
progress  of  software  development  efforts  (GAO,  2004). 

D,  CURRENT  F-22  SOFTWARE  DEVELOPMENT  STRATEGY  -  SPIRAL 
MODERNIZATION 

As  discussed  in  the  previous  chapter,  the  F-22  program  has  undergone  significant 
external  pressures  requiring  adaptation  to  changing  strategic  and  tactical  threats,  even 
while  fighting  for  funding  from  shrinking  DoD  budgets.  All  this  while  developing  one  of 
the  most  technically  complex  systems  fielded  to  date  by  DoD.  These  schedule  pressures, 
changing  requirements,  technical  risks,  and  funding  instabilities  haunted  the  F-22  EMD 
program  through  its  conclusion  in  2005.  With  the  need  to  deliver  unmet  EMD 
requirements  and  modernize  the  fighter  to  meet  emerging  threats.  Air  Eorce  leadership 
initiated  a  follow-on  effort  to  EMD  in  2003. 

Unlike  its  predecessor,  the  E-22  Modernization  Program  was  not  contracted  under 
one  behemoth  contract.  Rather,  it  was  contracted  with  Eockheed  Martin  Aeronautics 
(EM  Aero)  under  an  Indefinite  Delivery  Indefinite  Quantity  (IDIQ)13  contract  titled  the 
Raptor  Enhancement  Development  &  Integration  (REDI)  Contract.  The  REDI  contract, 
modeled  after  a  highly  successful  C-17  modernization  contract,  would  serve  as  the  single 

13  The  Eederal  Acquisition  Regulation  (EAR)  defines  an  IDIQ  as  “a  contract  for 
supplies  that  does  not  procure  or  specify  a  firm  quantity  of  supplies  (other  than  a 
minimum  or  maximum  quantity)  and  that  provides  for  the  issuance  of  orders  for  the 
delivery  of  supplies  during  the  period  of  the  contract”  (Subpart  16.501-1). 
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contract  for  “planning,  analysis,  design,  development,  qualifieation,  test  and 
doeumentation  of  performanee  enhaneements”  neeessary  for  the  F-22  mission  (F-22 
System  Program  Offiee,  2003).  Onee  awarded,  the  basie  IDIQ  eontraet  would  authorize 
work  via  individual  delivery  orders  foeused  on  speeifie  tasks  or  development  efforts.  The 
magnitude  of  the  delivery  orders  varied,  ranging  in  value  from  several  hundred  thousand 
dollars  to  several  hundred  million  dollars. 

The  first  delivery  order  (DO  0001)  awarded  under  the  new  REDI  eontraet  was  the 
System  Engineering/Program  Management  effort.  This  DO  was  the  starting  point  for  any 
enhaneement  eonsidered  for  the  P-22  and  eharged  EM  Aero  “to  provide  overall  Systems 
Engineering  and  Program  Management  in  support  of  the  P/A-22  program  to  maintain 
effeetive  ineorporation  of  ehanges  into  the  weapon  system”  (P-22  System  Program 
Offiee,  2003).  It  also  served  as  the  overarehing  arehiteeture,  aeeomplishing  all  early  and 
up-front  analysis  on  an  enhaneement  eandidate  before  eommitting  additional  resourees 
and  formally  proeeeding  with  a  stand-alone  delivery  order.  Therefore,  DO  0001  is  where 
Spiral  2  found  its  start. 

In  2002,  when  the  P-22  program  was  still  three  years  from  eompleting  EMD,  its 
users  began  looking  forward  to  what  the  fighter  jet  would  look  like  when  it  was  deelared 
mission  ready  at  IOC;  the  users  realized  it  would  fall  short  in  some  eapabilities.  As  these 
eapabilities  were  identified,  quantified,  and  prioritized,  a  list  began  to  emerge  that  would 
define  the  first  upgrade  to  the  jet  one  year  after  IOC  deelaration  in  Deeember  2005.  The 
upgrades  foeused  on  a  software-only  evolution  to  the  avionies  Operational  Plight 
Program.  Although  the  upgrade  was  later  identified  by  EM  Aero  in  more  aeeurate  terms 
as  Bloek  20,  the  government  organizations  and  doeumentation  eontinued  to  eall  the 
program  by  the  original  name,  a  name  that  eaptured  one  of  the  latest  buzz  words  in  DoD 
aequisition — Spiral  2.14 


14  Spiral  2  is  not  a  spiral  produet  as  the  name  would  imply.  While  eaeh  spiral  of  the 
P-22  modernization  program  built  on  the  preeeding  spirals  eapabilities,  the  upgrades 
more  elosely  resembled  inerements  versus  spiral  releases.  In  2004  eonsiderable 
diseussion  between  the  P-22  program  and  senior  Air  Poree  leadership  eentered  around  the 
eorreet  terminology  for  the  modernization  upgrades.  In  the  end  it  was  deeided  to 
eontinue  to  eall  them  spirals  while  they  would  be  managed  internally  by  government  and 
eontraetor  personnel  as  bloek  upgrades. 
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With  EMD  and  the  IOC  baseline  representing  Spiral  1,  Spiral  2  represented  the 
first  of  several  upgrades  planned  for  the  jet  in  the  modernization  program.  As  the 
pathfinder,  Spiral  2  established  the  proeedures  and  template  for  the  mueh  larger  and  more 
eomplex  software/hardware  upgrades  of  Spirals  3  and  4  that  would  follow.  In  Mareh 
2003,  within  the  seope  of  DO  0001,  requirements  analysis  for  Spiral  2  started  ereating  a 
list  of  potential  enhaneements  ealled  eandidates.  These  eandidates  were  further 
developed  and  earried  forward  to  the  end  of  this  phase  based  on  several  eonstraints:  user 
priority,  available  funding,  and  sehedule  alignment.  As  stated  above.  Spiral  2  was  a 
schedule  driven  upgrade  that  planned  to  deliver  software-only  upgrades  to  the  OFF  not- 
later-than  one  year  after  IOC — or  December  2006.  These  constraints — along  with 
funding  limitations — ^would  eventually  narrow  the  list  to  a  handful  of  approved 
candidates  to  carry  forward  to  the  follow-on  delivery  order.  Spiral  2  completed 
requirements  analysis  in  May  2004  and  was  ready  for  the  next  phase  of  the  program. 

Not  only  was  the  modernization  contract  broken  into  individual  contract  vehicles 
called  delivery  orders,  but  the  larger  spiral  upgrades  were  broken  into  different  delivery 
orders.  The  plan  developed  for  the  spirals  accomplished  requirements  analysis  on  DO 
0001.  Next  a  separate  delivery  order  was  awarded  for  the  detailed  design  of  the  upgrade. 
Upon  completion  of  the  design  effort,  another  delivery  order  would  be  initiated  to 
accomplish  the  coding,  integrating,  developmental  testing  and  post-operational  test 
updates  to  the  upgrade.  This  would  lead  to  a  modification  to  the  production  contract  that 
would  field  the  completed  product.  Although  this  process  resulted  in  tremendous 
pressures  on  the  business  processes  in  the  program,  it  afforded  the  program  the  flexibility 
to  adapt  to  the  pressures  that  caused  EMD  to  flounder  so  many  times.  As  funding 
realities  changed,  as  technology  challenges  were  realized,  and  as  requirement  priorities 
shuffled,  each  transition  between  delivery  orders  enabled  “on-ramps”  and  “off-ramps”  for 
capabilities. 

For  Spiral  2,  the  second  contract  effort  was  DO  0002.  This  contract  was  started  in 
March  2004  and  included  all  tasks  necessary  to  accomplish  Preliminary  Design  Review 
and  Critical  Design  Review  for  the  candidates  identified  in  the  contract.  This  was  the 
first  stand-alone  Spiral  2  delivery  order  and  utilized  EVM  as  a  management  tool  for  the 
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duration  of  the  contract.  Unfortunately,  the  fluidity  of  requirements  and  funding 
combined  with  the  lengthy  timelines  associated  with  awarding  REDI  delivery  orders  (up 
to  ten  months  from  solicitation  to  award),  forced  program  management  involved  to  award 
DO  0002  as  an  Undefinitized  Contract  Action  (UCA).  In  Chapter  IV,  F-22  EVMS 
Environment,  the  impacts  of  UCAs  on  a  program’s  EVM  will  be  discussed;  prior  to  that, 
however,  a  short  background  on  the  use  of  UCAs  seems  appropriate. 

Eack  of  funding  predictability,  emerging  technical  requirements  from  the  user, 
and  lengthy  business  review  processes  were  just  a  few  of  the  challenges  facing  each 
incremental  upgrade  for  the  F-22.  These  challenges,  combined  with  competition  for 
business  resources,  often  led  to  the  initiation  of  efforts  using  UCAs.  A  UCA  permits  the 
initiation  of  an  effort  without  a  firm  (definitized)  contract  in  place.  1 5 

While  this  paper  does  not  try  to  tackle  the  complex  issues  surrounding  the  pros 
and  cons  of  using  UCAs,  the  risks  associated  with  proceeding  under  undefinitized 
contracts  makes  this  method  a  contracting  tool  limited  to  those  instances  where  it  is 
absolutely  necessary.  Although  used  by  exception,  highly  scrutinized,  and  not  typically 
desirable,  in  the  F-22  program,  use  of  this  method  of  contract  award  has  become 
prevalent. 

Even  with  the  inherent  speed  of  a  UCA,  Spiral  2  DO  0002  was  completed  in 
February  2005 — two  months  past  the  original  period  of  performance.  The  completion  of 
the  detailed  design  cleared  the  way  for  initiation  of  the  software  coding  and  integration 
phase.  This  effort  was  awarded  in  January  2005  under  a  partial  UCA  for  DO  0019  on  the 
REDI  contract.  (The  remainder  of  the  effort  was  authorized  under  a  UCA  in  March 
2005.)  As  with  DO  0002,  this  effort  would  proceed  for  a  long  duration  under  a  UCA. 
Unlike  DO  0002,  it  would  be  definitized  (negotiated  via  a  firm  contract)  prior  to  its 
completion.  DO  0019  was  definitized  in  October  2005  and  is  expected  to  complete  in 
September  2007. 


15  The  Defense  Federal  Acquisition  Regulations  Supplement  (DEARS)  defines  an 
UCA  as  “any  contract  action  for  which  the  contract  terms,  specifications,  or  price  are  not 
agreed  upon  before  performance  is  begun  under  the  action”  (Subpart  2 17.740 Id). 


34 


E,  SUMMARY 

This  concludes  Chapter  II,  Literature  Review.  The  next  chapter,  F-22 
Implementation  of  ANSI/EIA  EVMS  Criteria,  uses  the  32  ANSI  eriteria  to  analyze  eaeh 
eriterion  separately  and  derive  a  eonelusion  regarding  the  degree  of  alignment  between  F- 
22  program  implementation  and  the  eriterion’ s  intended  purpose. 
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III.  F-22  IMPLEMENTATION  OF  ANSI/EIA  EVMS  CRITERIA 


A.  PREFACE 

The  32  criteria  identified  in  ANSI/EIA  748-1998  are  recognized  by  both  industry 
and  DoD  as  the  minimum  standards  for  establishing  a  useful,  functioning  EVMS. 
Meeting  these  criteria  is  critical  if  an  organization  is  going  to  be  able  to  successfully  use 
the  EVMS  as  a  management  control  tool.  In  order  to  assess  the  implementation  of  the 
EVMS  in  the  E-22  program,  these  criteria  were  used  to  evaluate  implementation  of  the 
EVMS  in  the  Spiral  2  modernization  program.  This  assessment  was  not  intended  to 
address  whether  EM  Aero  EVMS  policy  and  procedures  were  sufficient.  EM  Aero  has 
already  demonstrated  to  DoD  their  processes  comply  with  the  criteria  via  their 
certification  from  industry.  Instead,  this  was  a  subjective,  qualitative  assessment,  based 
on  the  criteria  objectives  outlined  by  ANSI/EIA,  data  and  procedures  gathered  from  the 
E-22  program  and  the  authors’  direct  observations,  which  looked  at  how  the  32  criteria 
were  actually  applied  through  the  implementation  of  Spiral  2  contracts. 

Eor  each  criterion  an  assessment  was  made  resulting  in  one  of  five  ratings: 
excellent,  satisfactory,  marginal,  insufficient  and  inconclusive.  Based  on  supporting  data 
that  exceeded  the  purpose  of  its  respective  ANSI  criteria,  the  following  seven  criteria 
were  conferred  an  excellent  rating:  4,  14-16,  19,  30,  and  31.  Adequate  supporting  data 
that  met  the  purpose  of  its  respective  ANSI  criteria,  led  to  the  following  15  criteria  being 
conferred  a  satisfactory  rating:  1-3,  5,  7,  10,  11,  13,  17,  22-25,  28,  and  32.  Due  to 
supporting  data  that  contained  minor  material  failures  related  to  meeting  the  purpose  of 
its  respective  ANSI  criteria,  the  following  five  criteria  were  conferred  a  marginal  rating: 
6,  8,  12,  18,  and  27.  One  criterion,  26,  was  conferred  an  insufficient  rating  based  on 
supporting  data  that  contained  more  than  minor  material  failures  regarding  its  respective 
ANSI  criteria,  the  following  criterion.  Einally,  an  inadequate  amount  of  supporting  data 
for  a  given  ANSI  criteria,  led  to  the  following  four  criteria  being  conferred  an 
inconclusive  rating:  9,  20,  21,  and  29. 
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The  next  seetion  eontains  a  more  detailed  diseussion  of  these  assessments  for  eaeh 
of  the  eriteria.  Specifieally,  eaeh  eriterion  ineludes  a  brief  overview  of  the  intent  of  the 
eriterion  (Fleming  and  Koppelman,  2000),  F-22  (LM  Aero)  polioy/proeedures  for 
meeting  the  eriteria  (Loekheed  Martin  Aeronauties  Corporation,  2005)  and  a  discussion 
of  the  authors’  assessment  for  each  criterion. 

B.  CRITERIA  ASSESSMENTS 

1,  Group  1:  Organization  Criteria 

Table  2  summarizes  the  assessments  provided  to  each  of  the  five  ANSI/EIA 
EVMS  Organization  Criteria. 


Criteria 

Assessment 

I  Define  authorized  work  (WBS  elements) 

Satisfactory 

2  Identify  organizational  responsibilities 

Satisfactory 

3  Integrate  the  system 

Satisfactory 

4  Identify  overhead  management 

Excellent 

5  Provide  for  performance  measurement 

Satisfactory 

Table  2.  Assessment  of  E-22  Spiral  2  Implementation  of  ANSI/EIA  EVMS  Organization 

Criteria 

a.  EVM  Criterion  #1:  Define  the  authorized  work  elements  of  the 
program.  A  WBS,  tailored  for  effective  internal  management 
control,  is  commonly  used  in  this  process. 

This  initial  criterion  addresses  the  necessity  of  starting  a  program  only 
after  fully  defining  its  requisite  efforts.  As  hinted  at  within  the  criterion  itself,  military 
acquisition  programs  require  a  program-specific  WBS  which,  through  its  very  nature, 
satisfies  this  criterion.  16  Consequently,  work  requested  by  a  customer  not  identifiable 
within  an  already  developed  WBS  should  be  considered  out-of-scope,  and  the  customer 
and/or  contracted  organization  should  seek  EAR-approved  authorization  methods  to 
begin  such  work  (in  this  case,  the  Air  Eorce  and  EM  Aero,  respectively). 


16  As  described  per  MIE  Handbook  881. 
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With  respect  to  Spiral  2-related  work,  LM  Aero  stated  that  tasks  were 
authorized  for  accomplishment  through  modifications  to  the  basic  Raptor  Enhancement 
Development  and  Integration  (REDI)  contract  (via  the  E-22  SPO’s  contracting  officer). 
Once  signed,  these  Delivery  Orders  (DOs)  then  energized  EM  Aero’s  Business 
Management  division  to  initiate  a  Sales  Order,  which  by  itself  serves  as  notification  of 
task  authorization  to  Spiral  2’s  management  team  at  Eockheed.  The  management  team 
analyzed  the  DOs,  Sales  Orders,  and  Statements  of  Work  (SOW)  within  proposals  for  the 
actual  allocation  of  authorized  tasks  to  their  appropriate  WBS  elements.  At  the  direction 
of  ASCAfEK  (the  customer  contracting  officer)  via  a  Contract  Data  Requirements  Eist 
(CDRL),  an  official  WBS  was  drafted. 

EM  Aero  communicated  that  the  WBS  was  structured  to  conform  to  the 
latest  version  of  MIE  Handbook  881  and  that  the  WBS  includes  all  Contract  Work 
Breakdown  Structure  (CWBS)  elements  specified  for  external  reporting  by  the  CDRL. 
Direct  observation  of  the  Cost  Performance  Reports  (CPRs)  supports  this  assertion.  The 
WBS  appeared  to  capture  all  authorized  work  due  to  the  lack  of  changes  to  the  WBS 
throughout  the  lifecycle  of  the  Spiral  2  program.  In  conclusion,  with  respect  to  LM 
Aero’s  actions  concerning  the  definition  of  authorized  work  for  the  Spiral  2  program,  by 
meeting  this  criterion  to  the  letter  this  EVMS  program  warrants  a  satisfactory  rating. 


b.  EVM  Criterion  #2:  Identify  the  program  organizational 
structure,  including  the  major  subcontractors  responsible  for 
accomplishing  the  authorized  work,  and  define  the 
organizational  elements  in  which  work  will  be  planned  and 
controlled. 

The  Organizational  Breakdown  Structure  (OBS)  must  be  established  to 
ensure  that  all  elements  of  the  WBS  (established  per  Criterion  #1),  are  assigned  to  a 
specific  organization  or  individual.  The  establishment  of  the  relationship  between  the 
OBS  and  WBS  results  in  an  OBSAVBS  assignment  matrix.  This  product  ensures  clearly 
defined  responsibility  for  each  task’s  completion. 

LM  Aero  establishes  their  OBS  according  to  an  Integrated  Product  Team 


(IPT)  structure.  Here  each  level  or  tier  has  IPTs  respectively  assigned  to  one  IPT  above 
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them,  ensuring  that  any  IPX  will  only  be  subordinate  to  one  IPX  (Figure  5  provides  an 
example  of  the  IPX  breakout  for  Spiral  2).  Xhis  approaeh  enables  one  element  of  the 
WBS  to  be  assigned  to  one  IPX  (or  major  subeontractor).  Xhe  interfaee  between  one 
organization  and  one  WBS  element  is  what  defines  a  eost/sehedule  account.  While  each 
WBS  element  may  only  be  assigned  to  one  organization,  multiple  WBS  elements  may  be 
assigned  to  any  one  organization. 


F-22  D00002  IPX  Structure 

Air  Vehicle 

Air  Vehicle  Systems 
Engineering  and  Integration 
Xeam  (SEIX) 

Air  Vehicle  Systems 

Build  Team 

Mission  System  &  Software 

Avionics  SEIX 
Core  Processing 
Communications,  Navigation 
&  IdentificationI 
Stores  Management  System 
Display  Products 
Electronic  Warfare 
Mission  Avionics  Software 
Radar 

Sustainment  &  Modifications 

Modifications  and  Heavy 
Maintenance 
Support  Equipment 
Support  Data 
Support  Services 
Xraining 

Weapon  System  Integration  & 
Technology 

Elight  Test 

Elight  Termination  System 
Test  Plan  &  Support 
System  Integration 

Program  Operations 
Modernization 

Eigure  5.  Spiral  2  IPX  Breakout 
(Prom:  Spiral  2  CPR  (Pormat  2),  2004) 
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LM  Aero  appropriately  established  a  detailed  OBS/WBS  assignment 
matrix  for  all  Spiral  2  eontract  efforts.  Although  the  initial  delivery  order  for 
Requirements  Analysis  Phase  did  not  establish  this  produet  in  as  mueh  detail  as  the  two 
follow-on  efforts,  it  did  provide  the  neeessary  relationship  between  the  WBS  elements 
and  organizations  assigned  to  eaeh  of  these  elements.  As  diseussed  above,  this  effort  was 
aoeomplished  as  part  of  an  overarehing  delivery  order  that  eneompassed  several 
modernization  efforts.  As  a  result,  the  WBS  elements  for  the  Spiral  2  effort  were  at  a 
higher  level  than  those  developed  and  assigned  for  the  two  Spiral  2  speeifie  delivery 
orders. 

Additionally,  a  eritieal  eoneern  with  this  eriterion  that  will  be  repeated  for 
many  other  eriteria  is  the  timeliness  of  the  establishment  of  the  OBS/WBS  assignment 
matrix.  The  Integrated  Baseline  Review  (IBR)'^  for  eaeh  of  the  three  Spiral  2  efforts 
was  held  signifieantly  later  than  the  initiation  of  the  eontraet  effort.  1 8  Based  on  the 
authors’  experienee,  this  was  driven  by  the  lack  of  a  definitized  contract,  changing 
government  requirements  and  immature  modernization  processes.  The  LM  Aero  cost 
accounting  process  made  the  creation  of  cost/schedule  accounts  mandatory  in  order  for 
work  to  proceed,  without  the  convening  of  an  IBR.  There  are,  however,  significant 
questions  regarding  whether  these  cost/schedule  accounts  were  established  in  a  planned 
and  controlled  manner,  a  manner  that  ensured  appropriate  development  of  relationships 
between  all  WBS  and  OBS  elements.  These  concerns  notwithstanding,  with  the  intent  of 
this  criterion  being  met  by  LM  Aero  actions  and  procedures  warrants  a  satisfactory 
rating. 


17  An  IBR  is  typically  held  within  the  early  stages  of  a  contract  period  of  performance 
and  it  “establishes  a  mutual  understanding  of  the  project  performance  measurement 
baseline”  and  provides  “an  agreement  on  a  plan  of  action  to  evaluate  the  risks  inherent  in 
the  program  measurement  baseline  and  the  management  processes  that  operate  during 
project  execution.”  (DAG,  4. 3. 2.4. 2) 

18  See  Chapter  IV,  Section  B  for  a  discussion  on  delayed  Spiral  2  IBRs. 
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c.  EVM  Criterion  #3:  Provide  for  the  integration  of  the  company’s 
planning,  scheduling,  budgeting,  work  authorization,  and  cost 
accumulation  processes  with  each  other,  and,  as  appropriate,  the 
program  WBS  and  the  program  organizational  structure. 

To  ensure  project  goals  are  given  priority  over  any  one  functional  area’s 
goals,  the  program  must  employ  an  integrated,  single  management  control  system  using 
common  information  from  the  programs  functional  areas.  The  integration  of  master 
scheduling,  cost  estimating,  work  authorizations,  budgeting  and  cost  accumulation  must 
work  within  a  single  database  to  ensure  managers  can  get  a  complete  picture  of  program 
health  and  make  management  decisions  based  on  inputs  from  all  functional  disciplines. 

The  LM  Aero  defines  an  integrated  process  for  developing  project 
schedules  and  budgets,  authorizing  the  work  associated  with  those  plans  and 
accumulating  and  reporting  costs  and  schedule  progress  consistent  with  the  established 
WBS  elements  of  the  contract.  During  the  development  of  the  project  schedule,  activities 
are  defined  along  with  their  interdependencies  with  other  activities.  These  activities  will 
become  the  basis  for  measuring  performance  in  terms  of  resource  requirements  (i.e.,  cost 
to  complete)  and  individual  work  package  progress.  Similarly,  budgets  are  developed, 
authorized  and  accumulated  by  individual  WBS  elements  (cost/schedule  accounts), 
allowing  measurement  of  program  health  at  both  the  discreet  work  package  level  and 
“rolled  up”  higher  WBS  levels.  This  “rolling  up”  of  program  cost  is  accomplished 
mechanically  and  allows  for  flexible  reporting  of  program  status  at  varying  project  levels 
based  on  the  desired  focus. 

The  integration  of  the  different  functional  areas  of  the  Spiral  2  WBS  and 
OBS  was  consistently  observed.  The  development  of  the  detailed  Integrated  Master 
Schedule  (IMS),  authorization  of  budget,  and  cost  accumulation  were  all  clearly  tied  to 
the  program  WBS.  Additionally,  organizational  responsibility  was  clearly  defined  via  the 
assignment  of  cost/schedule  accounts  to  individual  IPTs. 

One  concern,  that  will  be  addressed  later,  is  the  relationship  between  the 
project  IMS  and  the  detailed  IPT  activity  schedules  that  supported  the  resource-loaded 
IMS.  Specifically,  the  issue  was  how  well  they  were  linked  and  controlled.  IMS  and 
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associated  cost/schedule  accounts  were  resource  loaded  and  managed  per  defined  work 
authorizations  and  performance  measures.  Lower  tier  IPX  schedules  developed  to 
support  the  assigned  activities  were  not  directly  linked  to  the  IMS  and,  therefore, 
permitted  IPTs  to  “interpret”  their  detailed  schedules  and  take  credit  for  progress  against 
the  IMS  tasks.  This  approach  is  not  consistent  with  the  intent  of  this  criterion  which 
asserts  the  goal  of  measuring  all  progress  against  project  goals  versus  individual  IPX 
goals.  This  concern  does  not  affect  the  intent  of  this  criterion  enough  to  warrant  a 
negative  assessment;  therefore  a  satisfactory  rating  was  given  to  this  criterion. 

d.  EVM  Criterion  #4:  Identify  the  company  organization  or 
function  responsible  for  controlling  overhead  (indirect  costs). 

Adequate  identification,  allocation  and  tracking  of  program  indirect  costs 
is  a  concern  for  many  programs.  This  is  the  first  of  four  of  the  thirty-two  EVM  criteria 
that  deal  with  management  of  indirect  costs  (others  are  Criteria  #13,  #19,  and  #24). 
Although  not  directly  controlled  by  any  individual  project  manager,  indirect  costs  must 
still  be  clearly  identified  as  a  category,  formally  documented,  and  assigned  to  individual 
managers  responsible  for  authorization  and  control. 

Control  of  overhead  rates  and  application  to  a  specific  LM  Aero  contract 
is  the  responsibility  of  the  Overhead  Section  of  the  Aeronautics  Controller.  Surveillance 
of  overhead  costs  allocated  to  the  contract,  however,  is  the  responsibility  of  the  project 
and  functional  managers. 

Overhead  rates  applied  to  Spiral  2  contracts  were  used  by  LM  Aero  based 
on  rates  negotiated  with  Defense  Contract  Management  Agency  (DCMA).  DCMA  was 
also  responsible  to  review  performance  reports  to  ensure  rates  were  being  applied  in 
accordance  with  agreements.  Although  negotiation  of  each  Spiral  2  contract  focused  on 
the  applicability  and  appropriateness  of  applied  overhead  categories,  once  the  negotiated 
indirect  costs  were  authorized  per  the  negotiated  contract,  the  only  control  measure  for 
the  LM  Aero  and  government  program  managers  was  a  comparison  of  allocated  overhead 
to  the  overhead  portion  of  contract  budget.  LM  Aero  procedures  and  performance 
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regarding  the  management  of  overhead  indieated  an  excellent  rating  was  appropriate  for 
this  eriterion. 


e.  EVM  Criterion  #5:  Provide  for  integration  of  the  program  WBS 
and  the  program  organizational  structure  in  a  manner  that 
permits  cost  and  schedule  performance  measurement  by  elements 
of  either,  or  both,  structures  as  needed. 

In  order  to  measure  performanee,  a  standard  must  exist.  This  eriterion 
eoneerns  the  formation  of  that  standard,  known  within  the  military  aequisition 
eommunity  as  the  program  baseline.  This  eoneept  has  proven  so  important  that  three 
other  eriteria  will  deal  with  the  issue  of  implementing  a  baseline.  19  This  eriterion 
eoneerns  only  the  foundations  of  that  baseline — that  is,  the  integration  of  WBS  and  OBS. 

Aeeording  to  LM  Aero,  management  used  IPTs  to  integrate  Spiral  2’s 
WBS  and  organizational  struetures.  In  general,  IPTs  had  the  responsibility  of 
aeeomplishing  tasks  within  speeifie  CWBS  elements.  LM  Aero  allowed  that,  in  general, 
an  IPT  may  have  more  than  one  assigned  CWBS  element,  but  that  IPTs  did  not  share  a 
single  CWBS  element.  This  assignment  of  a  speeifie  CWBS  element  to  an  IPT 
established  the  eost/sehedule  aeeount,  the  base  level  of  eontrol  whieh  enabled  future  eost 
and  sehedule  measurements. 

The  proof  of  these  assertions  lay  in  the  faet  that  BCWS,  BCWP,  and 
ACWP  were  available  at  the  eost/sehedule  aeeount  level.  This  data  also  was  direetly 
summarized  to  only  one  higher  CWBS  and  only  one  IPT  strueture  element  in  the  CPR. 
Although  the  data  was  theoretieally  available  for  summarizing  at  any  strueture  level, 
reports  were  only  generated  at  the  detail  that  LM  Aero  was  eontraetually  obligated  to 
transmit.  In  the  ease  of  Spiral  2,  the  CDRL  ealled  for  three  reportable  tiers  within  the 
CPR.  In  eonelusion,  with  respeet  to  LM  Aero’s  notions  oonoerning  the  integration  of 
program  WBS  and  OBS  for  the  Spiral  2  program,  the  EVMS  program  warrants  a 
satisfactory  rating  for  meeting  the  letter  of  this  eriterion. 


19  ANSI/EIA  EVMS  Criteria  #17,  #18,  and  #25. 
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2,  Group  2:  Planning,  Scheduling  and  Budgeting  Criteria 

Table  3  lists  the  assessments  for  eaeh  of  the  ten  Planning,  Scheduling  and 
Budgeting  Criteria  in  Group  2  of  the  ANSI/EIA  EVMS  Criteria. 


Criteria 

Assessment 

6 

Schedule  the  work 

Marginal 

7 

Identify  products,  milestones  and  indicators 

Satisfactory 

8 

Plan  the  Performance  Measurement  Baseline  (PMB) 

Marginal 

9 

Establish  budgets  for  work 

Inconclusive 

10 

Identify  work  packages 

Satisfactory 

11 

Summarize  work  package  budgets  to  control  accounts 

Satisfactory 

12 

Identify  and  control  level  of  effort 

Marginal 

13 

Establish  overhead  budgets 

Satisfactory 

14 

Identify  management  reserves  and  undistributed  budget. 

Excellent 

15 

Summarize  budgets  to  target  cost 

Excellent 

Table  3.  Assessment  of  F-22  Spiral  2  Implementation  of  ANSI/EIA  EVMS  Planning, 

Scheduling  and  Budgeting  Criteria 


a.  EVM  Criterion  #6:  Schedule  the  authorized  work  in  a  manner 
that  describes  the  sequence  of  work  and  identifies  the  significant 
task  interdependencies  required  to  meet  the  requirements  of  the 
program. 

When  developing  the  IMS,  a  contractor  must  ensure  that  all  activities 
required  to  complete  the  effort  and  the  relationships  between  those  activities  are  well 
defined.  Many  programs  do  not  take  the  necessary  steps  to  develop  the  IMS  to  the 
necessary  level  of  detail.  In  order  for  EVM  to  be  a  useful  management  control  tool,  the 
IMS  must  be  accurate  with  respect  to  allocation,  consistency  and  traceability  of  budgeted 
schedule  and  resources. 

Eor  EM  Aero  the  process  of  developing  a  project  schedule  begins  with  the 
Contract  Delivery  Schedule  and  ends  with  fully  defined  activity  schedules.  The  factors 
affecting  the  scheduling  process  include  required  resources,  available  resources,  span 
times,  activity  relationships  and  external  constraints.  As  these  factors  are  considered, 
activities  are  defined  that  produce  necessary  interim  and  final  product(s)  of  the  contract 
effort.  As  activities  are  defined  and  assigned  to  work  packages  (which  are  in  turn 
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assigned  to  a  given  cost/schedule  aceount),  schedule  requirements  are  identified  for  each 
activity  and  resulting  start/completion  dates  can  be  identified  for  a  particular  work 
package  merely  by  identifying  the  start  date  for  its  earliest  activity  and  the  completion 
date  for  its  final  activity.  The  LM  Aero  process  also  highlights  the  critical  and  sometimes 
overlooked  step  of  ensuring  that  interdependencies  are  clearly  defined  and  understood  for 
project  work  packages  and  their  activities. 

This  is  one  of  the  criteria  significantly  impacted  by  the  F-22  culture  of 
changing  requirements.  While  LM  Aero  took  steps  to  build  and  manage  to  detailed  and 
linked  schedules,  the  frequent  impacts  of  changing  requirements  made  developing  a 
baseline  schedule  very  difficult.  This  culture  of  change  resulted  in  Contract  Delivery 
Schedules  that  did  not  provide  confidence  for  the  IPTs  charged  with  developing  the 
detailed  work  package  activity  schedules.  All  three  Spiral  2  contracts  were  initiated  and 
progressed  several  months  (up  to  50  percent  of  the  contract  period)  prior  to  the 
establishment  of  a  baselined  IMS.  This  was  especially  troubling  when  considering  the 
fact  that  exit  criteria  for  the  first  two  contracts  included  detailed  IMS  for  their  respective 
follow-on  contracts. 

The  program  recognized  the  need  to  have  a  detailed  schedule  prior  to  the 
initiation  of  a  contract;  however,  the  fluid  requirements  (as  well  as  modernization 
program  immaturity)  made  this  unattainable  for  all  three  Spiral  2  contracts.  As  discussed 
above,  once  the  detailed  IMS  was  defined,  it  was  noted  in  many  cases  there  weren’t 
mechanical  linkages  between  the  IMS  activities  and  the  detailed  activity  schedules  used 
by  the  IPTs.  The  approach  used  by  LM  Aero  in  the  case  of  Spiral  2  was  to  define  IMS 
activities  by  time  span  versus  the  detailed  IPT  tasks  that  would  actually  be  required  to  be 
performed  for  completion  of  the  respective  activity.  These  shortcomings  led  to  a 
marginal  rating  for  this  criterion  associated  with  scheduling  the  work. 
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b.  EVM  Criterion  #7:  Identify  physical  products,  milestones, 
technical  performance  goals,  or  other  indicators  that  will  be  used 
to  measure  progress. 

In  order  to  take  eredit  for  earned  value,  a  projeet  must  first  identify  the 
meaning  of  value.  This  eriterion  requires  the  eontraetor  identify  tangible  measures  for 
determining  how  mueh  value  has  been  earned  in  the  progress  of  the  effort.  In  the  ease  of 
software  development,  where  many  interim  produets  are  diffieult  to  measure,  this  is  one 
of  the  more  ehallenging  eriteria  to  meet. 

LM  Aero  poliey  is  lightly  defined  for  this  eriterion.  It  does  identify  the 
requirement  to  objeetively  measure  progress  based  on  eompletion  of  tangible  produets, 
but  it  does  not  provide  guidanee  on  how  to  determine  what  is  tangible.  Their  poliey  also 
states  that  “in  most  oases”  progress  will  not  be  refleoted  for  a  partioular  aotivity  until  the 
aotivity  has  been  assessed  as  oomplete  (Lookheed  Martin  Aeronautios  Corporation, 
2005). 

This  eriterion  is  one  in  whioh  the  authors  have  seen  government  oustomers 
lose  oonfidenoe  in  eontraetor  EVMS.  The  ohallenge  is  for  the  eontraetor  and  government 
to  agree  on  the  value  of  the  produots  identified  as  program  measures.  Additionally,  one 
oould  argue  not  all  tangible  products  are  necessarily  measurable  in  terms  of  value — 
particularly  in  the  case  of  software  development.  For  example,  when  a  lower  tier  IPT 
finishes  coding  a  software  product,  how  much  value  should  be  assigned  to  this  product 
before  it  has  been  fully  tested  in  an  integrated  fashion? 

While  many  interfaces  can  be  tested,  most  of  the  problems  in  software 
testing  occur  during  integration  testing  versus  unit  testing.  To  give  only  minimal  value  to 
the  product  prior  to  integrated  testing  might  undervalue  the  product  and  not  accurately 
reflect  the  progress  of  the  project.  However,  too  many  times  a  program  takes  too  much 
credit  for  completion  of  the  unit  and  later  shows  unfavorable  variances  when  integrated 
testing  identifies  unplanned  rework  for  the  product. 

Recognizing  these  challenges  in  Spiral  2,  LM  Aero  attempted  to  assign 
values  to  the  completed  interim  products  based  on  the  possibility  (likelihood!)  of 
problems  with  integrated  testing  that  would  require  rework.  The  determination  of  how 
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much  value  to  assign  to  these  interim  produets  and  how  mueh  budget  to  assign  to 
integration  test  and  rework  activities  was  a  proeess  that  required  past  experienee.  In  this 
area  LM  Aero  and  their  major  supplier,  Boeing,  both  had  tremendous  experience 
developing  fighter  aireraft;  however,  the  new  challenges  of  developing  the  first  fully 
integrated  avionies  system  eombined  with  the  years  of  software  ehallenges  seen  in  F-22 
EMD,  made  this  eriterion  one  that  required  speeial  foeus  from  the  government.  LM  Aero 
efforts  to  eireumvent  all  of  these  ehallenges  and  adhere  to  their  well-defined  proeedures 
in  this  area  of  identifying  produets,  milestones  and  indieators  yielded  a  satisfactory  rating 
for  this  eriterion. 


c.  EVM  Criterion  #8:  Establish  and  maintain  a  time-phased  budget 
baseline  at  the  control  account  level,  against  which  program 
performance  can  be  measured.  Initial  budgets  established  for 
performance  will  be  based  on  either  internal  management  goals 
or  the  external  customer-negotiated  target  cost,  including 
estimates  for  authorized  but  undefined  work.  Budget  for  far- 
term  efforts  may  be  held  in  higher-level  accounts  until  an 
appropriate  time  for  allocation  at  the  control-account  level.  On 
government  contracts,  if  an  over-target  baseline  is  used  for 
performance  measurement  reporting  purposes,  prior  notification 
must  be  provided  to  the  customer. 

The  “time -phased  budget  baseline”  mentioned  above  deseribes  the  distinet 
EVM  term  known  as  Program  Management  Baseline  (PMB).20  The  PMB  must  include 
all  authorized  work,  and  thus  beeomes  beholden  to  the  sueeess  of  most  of  the  previous 
eriteria.  Unless  a  given  program  has  a  well-defined  PMB  with  effective  management 
eontrol  systems  in  place,  that  program  has  little  to  no  chance  of  providing  useful  insight 
regarding  performanee  status  using  EVM  data. 

Aeeording  to  LM  Aero,  the  PMB  was  established  and  maintained  at  the 
eost/schedule  aeeount  level.  Eor  Spiral  2  the  budget  was  based  on  the  negotiated  target 
eost,  to  inelude  estimates  for  any  eontraetually  authorized  but  not  negotiated  ehanges 
(e.g..  Undistributed  Budget).  The  seetion  of  this  eriterion  eonceming  reporting  PMBs 


20  See  discussion  within  the  EVM  Primer  seetion  of  Chapter  I. 
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reflecting  an  overrun  (the  “over-target  baseline”)  was  rendered  non  applicable  due  to  the 
fact  that  Spiral  2  experienced  an  underrun.  However,  LM  Aero  communicated  that 
before  any  of  their  projects  can  implement  an  over-target  baseline  for  PMB  purposes,  the 
LM  Aero  F-22  program  manager  and  Director  of  Cost  Management  Integration  must 
justify  and  provide  prior  notification  to  the  customer — the  F-22  SPO. 

Evaluating  F-22  program  implementation  of  this  criterion  necessitates 
dividing  it  into  two  parts:  establishment  and  maintenance.  The  establishment  half  of  this 
criterion  warrants  a  satisfactory  rating,  given  1 .)  the  establishment  of  the  PMB  within  the 
Spiral  2  CPRs  and  Contract  Funds  Status  Reports  (CFSRs);  2.)  the  CPR  and  CFSR 
reconciliations  with  respect  to  authorized  budgets;  3.)  the  CPR  and  CFSR  reconciliations 
between  the  two  reports  themselves.  The  maintenance  half  of  this  criterion,  much  like 
Criterion  #6,  was  significantly  impacted  by  the  F-22  culture  of  changing  requirements. 
Specifically,  late  contract  definitization  (and  subsequent  IMS  baselining)  rendered  the 
reported  EVM  data  virtually  useless. 

As  also  described  in  Criterion  #6,  the  approach  used  by  EM  Aero  in  the 
case  of  Spiral  2  was  to  define  IMS  activities  by  time  span  versus  the  detailed  IPT  tasks 
actually  required  to  be  performed  for  completion  of  the  respective  activity. 
Consequently,  the  final  CPR  reflected  a  PMB-derived  EAC  of  $46M  even  while  noting  a 
“most  likely”  EAC  of  $24  million.  Thus,  up  to  the  point  of  the  final  CPR  submission  on 
January  2005,  all  EVM  data  reflected  performance  based  on  a  benchmark  192  percent 
more  than  the  yet- to-be  definitized  EAC.  As  a  result,  the  maintenance  half  of  this 
criterion  warrants  an  insujficient  rating.  In  conclusion,  with  respect  to  Spiral  2 
establishment  and  maintenance  of  a  time-phased  budget  baseline  at  the  control  account 
level,  the  EVMS  program  warranted  an  overall  marginal  rating  for  meeting  the  purpose 
of  this  criterion. 
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d.  EVM  Criterion  #9:  Establish  budgets  for  authorized  work  with 
identification  of  significant  cost  elements  (labor,  material,  and  so 
on)  as  needed  for  internal  management  and  for  control  of 
subcontractors. 

Criterion  #9  pertains  to  total  projeet  budgeting,  whieh  a  program  can  only 
accomplish  through  a  comprehensive  list  of  cost  elements.  As  the  above  alludes  to, 
effective  formal  control  systems  must  accompany  the  roll-up  of  cost  elements.  Budgeted 
values  must  equate  to  negotiated  project  costs,  from  the  standpoint  of  both  supply  (sub¬ 
contractors)  and  demand  (F-22  SPO). 

From  LM  Aero’s  perspective,  work  packages  and  planning  packages  were 
budgeted  by  elements  of  cost.  Specifically,  discrete  portions  of  the  total  contract  budget 
base  were  allocated  to  each  Cost/Schedule  Account  Manager  (C/SAM)  through  the 
Budget  Ledger.  LM  Aero  states  that  subcontracted  CWBS  elements  were  identihed 
within  the  accounting  system  by  “unique  work  orders  and  work-in-process  subaccounts” 
(Lockheed  Martin  Aeronautics  Corporation,  2005).  Within  the  EVMS,  these  accounting 
system  data  items  were  represented  by  specific  cost  element  codes.  Once  again,  the  use 
of  a  shared  CWBS  by  ah  elements  of  the  project  organization  assures  a  common 
understanding,  consistency  for  planning  and  performance,  and  effective  oversight  of  ah 
contractually  authorized  tasks. 

In  practice,  two  issues  surfaced  regarding  this  criterion.  First  and 
foremost  was  a  lack  of  verifiable  supporting  data.  As  explained  within  the  preface  of  this 
chapter,  the  above  paragraph  describes  policy  as  opposed  to  implementation.  It  should 
not,  therefore,  warrant  consideration  as  proof  positive  of  an  effective  execution  of  this 
criterion. 

The  second  area  of  concern  stemmed  from  the  description  of  the  Budget 
Ledger.  According  to  LM  Aero’s  own  policy,  the  ledger  may  authorize  the  budget  by 
CWBS  element  not  only  in  terms  of  total  dollars,  but  also  in  terms  of  less  discrete  cost 
elements  such  as  labor  hours/dohars  and  burden  overhead  dollars  (to  name  a  few 
examples).  If  the  Budget  Ledger  described  a  C/SAM  account's  budget  by  cost  element, 
responsibility  laid  with  the  IPX  leader  for  translation  of  said  cost  element  into  total  dollar 
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terms.  The  resultant  unclear  “flexibility  to  budget  the  work  packages/planning  packages 
of  the  cost/schedule  account  in  whatever  mix  of  resources  deemed  appropriate”  was  seen 
as  a  material  weakness  with  respect  to  this  criterion  (Lockheed  Martin  Aeronautics 
Corporation,  2005).  However,  that  concern  also  lacked  specific,  verifiable  supporting 
data.  As  such,  with  respect  to  Spiral  2  establishment  of  budgets  for  authorized  work  as 
needed  for  internal  management  and  for  control  of  subcontractors  and  maintenance  of  a 
time-phased  budget  baseline  at  the  control  account  level,  the  EVMS  program  warranted 
an  inconclusive  rating  due  to  lack  of  supporting  data. 

e.  EVM  Criterion  #10:  To  the  extent  that  it  is  practical  to  identify 
the  authorized  work  in  discrete  work  packages,  establish  budgets 
for  this  work  in  terms  of  dollars,  hours,  or  other  measurable 
units.  Where  the  entire  control  account  is  not  subdivided  into 
work  packages,  identify  the  far-term  effort  in  larger  planning 
packages  for  budget  and  scheduling  purposes. 

This  criterion  further  expands  on  Criterion  #8,  establishment  of  a 
definitive  PMB,  by  identifying  the  need  for  discreetly  defined  work  packages.  Although 
“far-term”  is  not  defined,  it  is  recognized  that  at  some  point  it  becomes  non-value-added 
to  attempt  discrete  definition  of  work  packages  that  are  too  far  removed  from  the  current 
state  of  the  program.  This  criterion  also  identifies  the  requirement  to  establish 
measurable  metrics  for  assessing  the  amount  of  work  accomplished  at  any  point  in  the 
effort. 

LM  Aero  policy  does  not  stipulate  the  delineation  between  near-term  and 
far-term  activities.  It  does,  however,  provide  the  planners  more  guidance  than  the 
ANSI/EIA  criterion.  While  EM  Aero  directs  that  “all  work  [be]  planned  for  the  duration 
of  the  contracf’  (Eockheed  Martin  Aeronautics  Corporation,  2005,  pi 2),  it  also 
recognizes  the  lack  of  certainty  or  definition  that  may  exist  in  activities  planned  outside 
of  the  current  year  of  effort.  The  general  approach  is  current  year  activities  will  be  part 
of  well-defined  work  packages  while  out  year  activities  will  be  assigned  to  planning 
packages.  It  is  important  to  note  that  while  there  exists  less  detail  associated  with  the 
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planning  packages,  they,  like  the  work  paekages,  are  still  defined  with  planned 
start/fmish  dates,  quantity  (units,  hours,  etc.)  and  dollars  of  resource  required. 

Due  to  the  relatively  short  duration  of  all  three  Spiral  2  eontraet  periods, 
LM  Aero  was  able  to  develop  disereetly  defined  work  paekages  for  the  majority  of  the 
work  performed  on  eaeh  effort.  The  first  two  eontraet  efforts  were  each  approximately 
one  year  in  duration.  The  third  was  just  over  two  years.  The  primary  challenge  of  this 
eriterion  was  the  ability  to  define  disereet  work  packages  in  an  environment  of  ehanging 
requirements.  With  all  three  of  these  efforts  proeeeding  under  a  UCA  eontraet,  there 
existed  a  laek  of  eertainty  regarding  the  eontent  of  the  eontraet  effort  and  eontraet 
funding.  This  led  to  more  work  than  desired  being  held  in  either  planning  paekages  or 
undistributed  budget  (see  Criterion  #14).  In  eonelusion,  LM  Aero  implementation  of  this 
eriterion,  identifying  the  need  for  disereetly  defined  work  paekages,  resulted  in  a 
satisfactory  rating. 

f.  EVM  Criterion  #11:  Provide  that  the  sum  of  all  work-package 
budgets,  plus  planning-package  budgets  within  the  control 
account,  equals  the  control-account  budget. 

The  sum  of  all  Spiral  2  work  and  planning-paekage  budgets  should  have 
been  equal  to  their  respeetive  control  account  budgets.  Furthermore,  eaeh  of  the  eontrol 
aeeount  budgets  must  have  been  related  to  a  speeifie  SOW.  The  only  aeeount  that  was 
not  ineluded  was  the  Management  Reserve  (MR)  aeeount,  held  in  general  outside  the 
purview  of  the  performanee  baseline. 

LM  Aero  acknowledged  that  distribution  of  Spiral  2-negotiated  target  eost 
was  made  from  the  individual  eost/sehedule  aeeounts  (aka  eontrol  aeeount)  to  their 
respective  work  and  planning-paekages.  In  all  cases,  LM  Aero  ensured  that  the  sum  of 
the  budgets  assigned  to  these  packages  equated  to  the  total  dollar  budget  authority  of  the 
eost/sehedule  aeeount.  Fulfillment  of  Criterion  #1  made  eertain  that  eaeh  of  the  eontrol 
aeeount  budgets  related  to  a  speeifie  SOW. 

The  CPRs  generated  in  support  of  Spiral  2  reporting  substantiated  LM 
Aero’s  poliey  regarding  this  eriteria.  An  assessment  of  the  data  showed  all  work 
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packages  correctly  summed  up  to  their  respective  control  accounts  throughout  the  period 
of  reporting  (Apr  2004  through  Jan  2005).  MR  grew  during  the  reports,  as  expected 
given  the  at-the-time  anticipated  underrun.  In  conclusion,  with  respect  to  Spiral  2 
summing  of  all  budgets  within  a  control  account  equating  to  the  authorized  total  of  that 
control  account,  the  EVMS  program  warranted  a  satisfactory  rating  for  meeting  the 
purpose  of  this  criterion. 


g.  EVM  Criterion  #12:  Identify  and  control  level-of-effort  (LOE) 
activity  by  time-phased  budgets  established  for  this  purpose. 
Only  that  effort  that  is  unmeasurable,  or  for  which  measurement 
is  impractical,  may  be  classified  as  LOE. 

LOE  activities  are  of  no  benefit  to  a  manager  using  EVMS  because  they 
measure  the  passage  of  time  versus  the  accomplishment  of  tasks  or  delivery  of  products. 
While  some  activities  clearly  fall  in  the  realm  of  LOE,  minimizing  the  categorization  of 
activities  as  LOE  is  necessary  for  a  manager  to  accurately  measure  and  manage  his 
program’s  health. 

EM  Aero  provides  little  guidance  in  the  area  of  controlling  LOE.  They 
merely  state  that  it  “will  exist  only  for  those  tasks  where  discreet  or  apportioned  work 
measurement  techniques  cannot  be  effectively  applied”  and  “will  be  separated  from 
discreet  and  apportioned  effort  at  the  work  package  level”  (Lockheed  Martin  Aeronautics 
Corporation,  2005). 

This  limited  discussion  of  LOE  and  lack  of  specific  guidance  was  apparent 
when  assessing  EM  Aero’s  approach  to  applying  LOE  as  a  work  measurement  approach 
for  Spiral  2  contracts.  While  it  was  expected  that  some  efforts  such  as  scheduling, 
configuration  management  and  program  management  activities  would  naturally  be 
associated  with  LOE,  the  use  of  LOE  in  Spiral  2  contracts  did  not  appear  to  be  in  line 
with  this  criterion’s  goal  of  minimizing  its  use.  One  example  was  the  use  of  LOE  for 
some  rework  activities  associated  with  the  development  of  requirements  and  design 
documentation  during  the  first  two  Spiral  2  contracts.  Because  of  the  difficulty  of 
defining  the  activity  associated  with  the  rework  of  documentation  during  review  cycles. 
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LOE  was  used  to  capture  this  effort.  This  is  just  one  example  where  LOE  could  have 
been  better  defined  as  discreet  work,  indicating  that  EM  Aero  could  better  focus  their 
EVMS  on  minimizing  EOE.  Examples  such  as  this  led  to  a  marginal  rating  for  this 
criterion  associated  with  control  of  EOE. 

h.  EVM  Criterion  #13:  Establish  overhead  budgets  for  each 
significant  organizational  component  of  the  company  for 
expenses  that  will  become  indirect  costs.  Reflect  in  the  program 
budgets,  at  the  appropriate  level,  the  amounts  in  overhead  pools 
that  are  planned  to  be  allocated  to  the  program  as  indirect  costs. 

This  criterion  highlights  concerns  regarding  the  proper  allocation  of 
indirect  costs  to  a  project  or  program.  To  preclude  manipulation  a  company  must  specify 
areas  of  indirect  cost  at  program  inception,  with  formal  internal  controls  directing  any 
subsequent  changes.  When  indirect  costs  do  not  allocate  directly  to  control  accounts,  the 
contracted  organization  should  indicate  some  point  within  the  WBS  where  the  indirect 
costs  apply. 

EM  Aero  policy  stipulated  a  formal,  annual  establishment  of  overhead 
budgets  plant-wide.  To  start  with,  EM  Aero  forecasted  both  known  and  estimated 
business  for  their  next  fiscal  cycle.  This  business  included  integration  of  the  annual 
overhead  budget  with  plans  for  contract  performance,  sales  and  profits,  capital 
investments,  and  cash  flow  requirements.  The  Overhead  Section  of  the  Aeronautics 
Controller  would  then  internally  publish  the  indirect  manpower  and  dollar  targets 
necessary  to  support  the  assumptions  used.21  Eollowing  that,  overhead  budgets  were 
established  with  functional  organization  development  of  internal  assessments  of 
requirements  Targets  were  developed  based  on  historical  trends,  current  spending  levels, 
expected  or  known  changes  in  task/requirements  and  other  quantitative  or  qualitative  data 
and  assigned  based  on  the  Aeronautics  Controller  Overhead  Section’s  assessment. 

Discrete  items  of  cost  were  assigned  to  identified  organizations  for 
planning  and  control  (starting  with  indirect  manpower)  and  formally  and  discretely 

21  The  assumptions  used  for  overhead  planning  were  not  made  available. 
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identified  to  the  appropriate  burden  eenter/  overhead  pool  (see  Figure  6  for  a  list  of 
typically  allocated  overhead  expenses;  see  Figure  7  for  a  generic  example  of  how  discrete 
expense  accounts  are  allocated  to  overhead  pools  at  LM  Aero).  Once  approved,  the 
budgeting  system  allowed  for  adjustments  to  the  overhead  budget  due  to  anticipated 
changes  in  conditions  and/or  assumptions.  Any  overhead  budget  adjustments,  however, 
required  formal  requests  “from  proper  line  management”  (Lockheed  Martin  Aeronautics 
Corporation,  2005).  Properly  requested  adjustments  required  detailed  justification,  which 
were  evaluated  by  the  Aeronautics  Controller.  LM  Aero  concluded  with  the  statement 
that  “appropriate  line  and  company  managers”  must  approve  all  upward  adjustments 
(Lockheed  Martin  Aeronautics  Corporation,  2005). 


LM-Aero  Overhead 
Accounts 

Salaries  &  Wages 
Employee  Awards 
Group  Insurance  Program 
Payroll  Taxes  &  Insurance 
Fringe  Benefits 
Retirement  Plan 
Savings  Plan 
Misc.  Employee  Benefits 
Indirect  Supplies  &  Material 
Utilities 

Depreciation  &  Amortization 
Taxes 

Travel  &  Communication 
Expense 

Miscellaneous  Expenses 
Proposal  &  Bidding  Expense 
Independent  Research  & 

Development 
Intercompany  Expense 

Figure  6.  Summary  of  Company  Overhead  Expense  Accounts 
(From:  Lockheed  Martin  Aeronautics  Company  EVMS  Description,  2005) 
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Figure  7.  Overhead  Pool  Assignment  for  Organizational  Expense 
(From:  Loekheed  Martin  Aeronautics  Company  EVMS  Description,  2005) 


In  practice,  the  ability  to  manipulate  indirect  costs  between  profitable  and 
less  profitable  programs  creates  a  heightened  concern  to  government  procurement 
offices.  The  Company’s  annual  publishing  of  its  EVMS  Description  and  Cost 
Accounting  Standards  created  the  expectation  that  EM  Aero  adhered  to  industry- 
acceptable  methods  of  indirect  cost  allocation.  Moreover,  tracking  and  auditing  for  the 
allocation  and  accumulation  of  indirect  costs  against  F-22  contracts  was  managed  by 
DCMA  located  in  the  EM  Aero  facilities  in  Fort  Worth,  Texas.  DCMA  is  responsible  for 
negotiating  and  monitoring  overhead  rates  for  all  EM  Aero  contract  efforts  based  in  Fort 
Worth,  including  Spiral  2  software  development.  Additionally,  they  received  and 
monitored  all  EVMS  reports  for  the  Spiral  2  effort,  and  identified  no  negative  findings  or 
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inconsistencies  with  how  LM  Aero  alloeated,  aoeumulated  or  reported  indirect  costs  on 
the  Spiral  2  contraets.  As  sueh,  with  respeet  to  Spiral  2’s  alloeation  of  indireet  eosts 
towards  authorized  budget,  the  EVMS  program  warranted  a  satisfactory  rating  for 
meeting  the  purpose  of  this  eriterion. 

L  EVM  Criterion  #14:  Identify  management  reserves  and 
undistributed  budget. 

Identifying  and  eontrolling  both  management  reserve  (MR)  and 
undistributed  budget  (UB)  is  neeessary  to  maintain  the  integrity  of  a  program’s  EVMS. 
MR,  used  to  eover  the  eost  of  “unknowns”,  must  be  held  outside  the  PMB  and  will  not  be 
assigned  to  a  WBS  element  until  a  deeision  is  made  by  management  to  do  so.  ElB,  part  of 
the  project  PMB,  represents  funds  that  have  been  identified  as  essential  for  eompletion  of 
the  projeet,  but  have  yet  to  be  assigned  to  a  WBS  element. 

While  EM  Aero  poliey  leaves  it  to  the  diseretion  of  the  program  manager 
whether  MR  is  required  for  a  project,  it  is  clear  in  direeting  that  MR  and  ElB  be 
maintained  separate  from  the  PMB.  Eurther,  it  elearly  stipulates  that  all  transactions  with 
respect  to  MR  and  UB  will  be  doeumented. 

All  three  Spiral  2  eontraets  had  both  MR  and  UB  elearly  identified  at  the 
IBRs.  With  all  three  efforts  being  initiated  as  UCAs  there  was  UB  for  all  three  eontraets. 
Once  the  efforts  were  negotiated,  the  UB  was  appropriately  distributed  to  the  applieable 
WBS  elements.  The  assessment  for  this  eriterion  assoeiated  with  MR  and  UB  was 
determined  to  be  excellent. 

j.  EVM  Criterion  #15:  Provide  that  the  program  target  cost  goal  is 
reconciled  with  the  sum  of  all  internal  program  budgets  and 
management  reserves. 

This  eriterion  coneentrates  on  the  aeeountability  of  all  projeet  funds.  As  a 
result,  a  eontractor  must  strive  to  keep  its  total  projeet  eosts  within  their  authorized 
budgets.  Along  those  same  lines,  the  eontraetor  must  exhibit  doeumented  eontrol 
proeesses  to  ensure  that  total  projeet  eosts  are  kept  in  eheek. 
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LM  Aero’s  response  to  this  eriterion  meets  expeetations.  The  negotiated 
contraet  target  eost  was  distributed  to  the  control  accounts — $36M  total.  The  total  target 
cost  ($46M)  also  included  UB  ($5M)  and  MR  ($5M).  Therefore,  the  amount  distributed 
to  the  control  accounts,  plus  the  value  of  UB  and  MR,  reconciled  to  the  contract's  total 
target  cost. 

Today’s  cost  tracking  software  makes  it  difficult — if  not  impossible — to 
fail  in  observing  this  criterion.  When  the  F-22  SPO  received  CPRs  from  LM  Aero 
electronically,  the  CPRs  were  accompanied  with  software  packets  that  ported  directly 
into  a  program  made  expressly  for  tracking  EV  data.  Therefore,  should  any  portion  of 
program  costs  not  equate  to  the  total,  the  software  identified  and  isolated  the  cost(s)  in 
question.  This  made  reconciliation  more  of  a  technical  issue  rather  than  an  analytical 
one — in  most  cases.  In  conclusion,  with  respect  to  Spiral  2’s  reconciliation  of  all  internal 
program  budgets  (plus  reserves)  the  EVMS  program  warranted  an  excellent  rating  for 
meeting  and  exceeding  the  purpose  of  this  criterion. 


3,  Group  3:  Accounting  Criteria 

Table  4  identifies  the  assessments  provided  to  each  of  the  six  ANSI/EIA  EVMS 
Accounting  Criteria. 


Criteria 

Assessment 

16  Record  direct  costs 

Excellent 

17  Summarize  direct  cost  to  the  WBS 

Satisfactory 

1 8  Summarize  direct  cost  to  the  organization 

Marginal 

19  Record  indirect  costs 

Excellent 

20  Identify  unit/lot  costs 

Inconclusive 

21  Record  material  costs 

Inconclusive 

Die  4.  Assessment  of  E-22  Spiral  2  Implementation  of  ANSI/EIA  EVMS  Accounting 

Criteria 


58 


a.  EVM  Criterion  #16:  Record  direct  costs  in  a  manner  consistent 
with  the  budgets  in  a  formal  system  controlled  by  the  general 
books  of  account. 

According  to  Fleming  and  Koppelman  (2000,  pl71),  the  preferred  method 
for  recording  direct  costs  is  “applied  direct  method”,  aecounting  for  resources  as  they  are 
used  or  eonsumed.  In  labor  intensive  efforts  sueh  as  software  development,  where  teams 
are  typically  established  by  functional  disciplines,  the  challenge  is  to  ensure  direct  labor 
costs  are  appropriately  eharged  to  the  eorrect  project  with  many. 

LM  Aero  policy  provides  great  detail  regarding  the  cost  identification  and 
numbering  systems,  work  order  nomenclature,  direet  charge  polieies,  work-in-process 
(WIP)  subaccounts,  recurring  vs.  nonrecurring  costs,  and  the  direct  labor 
charge/accounting  process.  LM  Aero  uses  the  applied  direet  method,  based  on  the 
proeedures  outlined  in  LM  Aero  policy. 

Although  not  unique  to  F-22  Spiral  2  software  development  (or  even  to 
software  development  in  general),  the  government  has  always  had  concerns  in  the  area  of 
cost  accounting  and  how  charges  are  traeked  to  different  work  packages.  With  managers 
or  IPTs  having  more  than  one  charge  account  at  their  disposal,  on  any  given  day,  what 
prevents  an  individual  or  IPX  from  charging  to  a  “healthy”  account  versus  the  one  they 
are  working  on  that  is  near  or  at  an  over-run  state?  Although  this  is  not  addressed  in  LM 
Aero  policy,  the  government  must  continue  to  rely  on  the  power  of  audits,  DCMA,  and 
LM  Aero  adherence  to  generally  aceepted  aeeounting  standards.  The  detail  provided  by 
LM  Aero  procedures  along  with  observed  implementation  warranted  an  excellent  rating 
for  this  criterion  associated  with  reeording  direet  costs. 

b.  EVM  Criterion  #17:  When  a  WBS  is  used,  summarize  direct 
costs  from  control  accounts  in  the  WBS  without  allocation  of  a 
single  control  account  to  two  or  more  WBS  elements. 

The  purpose  of  this  criterion:  to  eliminate  the  confusion  resulting  from 
multiple  WBS  elements  crisserossed  with  multiple  eontrol  accounts.  A  WBS  element  by 
its  nature  identifies  a  unique  eontrol  account,  which  eliminates  the  possibility  of  dividing 
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a  control  account  between  multiple  WBS  elements.  Additionally,  the  WBS  design 
ensures  that  a  lower  level  element  uniquely  identifies  with  one  (and  only  one)  higher 
level  element  or  tier. 

The  LM  Aero  F-22  team  strived  to  meet  this  eriterion  via  its  basie 
aceounting  numbering  system.  An  eight  eharaeter  work  order  forms  the  foundation  of  the 
system.  Spiral  2  work  orders,  through  their  unique  eight  eharaeters,  reeorded  and 
identified  ineurred  costs  to  the  contract,  then  to  the  eontraet  line  item,  followed  by  work 
breakdown  structure  elements,  and  finally  to  the  diserete  tasks  within  the  WBS  elements. 
Work  order  numbers  were  established  by  the  Aeeounting  Department  within  the  terms  of 
the  Spiral  2  contract  modifications  to  the  REDI  eontraet.  A  record  of  all  authorized  work 
orders  was  maintained  by  the  Accounting  Department,  while  active  work  orders  were 
maintained  in  a  computer  file  which  was  readily  accessible  by  company  personnel. 

In  practiee,  LM  Aero  identified  the  first  three  characters  of  the  work  order 
as  representing  the  eontraet  code  number  assigned  to  the  contract.  With  respect  to  Spiral 
2,  this  code  was  shared  with  all  REDI  contract  actions  and  subsequent  modifications. 
The  seeond  grouping  of  the  work  order  (eharaeters  four  and  five),  also  known  as  the 
project  code,  aligned  eosts  with  CWBS  elements.  The  third  and  final  grouping  of  the 
work  order  (eharaeters  six  through  eight),  also  known  as  the  job  code,  provided  detail 
within  the  projeet  by  identifying  eosts  along  with  their  respeetive  job  or  specifie  task 
item.  With  respeet  to  Spiral  2’s  WBS  utilization  and  respective  eontrol  aecounts  related 
to  one  and  only  one  given  WBS,  the  EVMS  program  warranted  a  satisfactory  rating  for 
meeting  the  purpose  of  this  criterion. 


c.  EVM  Criterion  #18:  Summarize  direct  costs  from  the  control 
accounts  in  the  contractor’s  organizational  elements  without 
allocation  of  a  single  control  account  to  two  or  more 
organizational  elements. 

This  eriterion  builds  on  Criterion  #2  whieh  identified  the  need  for 
assigning  eaeh  element  of  the  WBS  to  an  organization.  Criterion  #18  requires  a 
eontractor’s  eost  aeeounting  system  be  eapable  of  collecting,  summing  and  reporting  cost 
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accounts  by  functional  organizations  (e.g.,  engineering,  quality,  production,  etc.).  This 
provides  managers  an  ability  to  measure  the  work  being  aeeomplished  by  functional  area 
as  a  program  progresses.  This  eriterion  also  explicitly  stipulates  what  is  inferred  in 
Criterion  #2 — that  a  single  eontrol  account  may  not  be  assigned  more  than  one  functional 
organization. 

As  with  most  of  today’s  contractors,  LM  Aero  does  most  of  their 
development  activities  through  multi-funetional  IPTs.  Although  this  eriterion  appears  to 
eall  for  breaking  eost/schedule  aeeount  work  paekages  apart  based  on  funetional 
organizations,  LM  Aero  poliey  does  not  take  this  approaeh.  In  order  to  ensure  the  ability 
to  summarize  eosts  by  funetional  organizations,  LM  Aero  developed  an  employee 
numbering  approaeh  that  identifies  eaeh  employee  by  their  funetional  area  regardless  of 
what  organization  to  whieh  they  are  assigned.  Sinee  employees  reeord  their  hours  in  a 
work  package  using  their  employee  number,  a  summary  of  labor  eosts  can  be 
aeeomplished  by  functional  area. 

Spiral  2  used  many  funetional  diseiplines  during  its  development.  The 
ability  to  identify/summarize  effort  by  funetional  organization  was  never  observed  by  the 
government.  Although  LM  Aero  poliey  identifies  this  as  a  capability,  reports  showing 
this  capability  were  never  produeed.  Government  requests  for  these  reports  were  never 
supported  leaving  questions  regarding  LM  Aero’s  ability  to  meet  this  eriterion.  Based  on 
the  data  provided,  however,  there  was  enough  eoneern  to  warrant  a  marginal  rating  for 
this  eriterion. 


d.  EVM  Criterion  #19:  Record  all  indirect  costs  that  will  be 
allocated  to  the  contract 

EVM  Criterion  #19  is  related  to  the  funetional  responsibility  for 
eontrolling  indireet  eosts  of  Criterion  #4.  This  eriterion,  however,  goes  further  in 
requiring  that  the  eontraetor  be  able  to  identify  indireet  eosts  at  the  point  eharged, 
summarize  them,  and  relate  them  to  original  planned  budgets.  It  also  requires  the 
relationship  be  formally  doeumented  between  those  eontrolling  indireet  eosts  and  those 
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able  to  incur  costs  against  indirect  budgets.  Whatever  method  is  chosen  by  the  contractor 
to  allocate  indirect  costs,  it  must  be  documented,  consistently  applied  and  auditable. 

LM  Aero  policy  regarding  the  allocation  and  accumulation  is  consistent 
with  generally  accepted  accounting  procedures  and  uses  a  monthly  adjusted  year-to-date 
approach  that  is  intended  to  minimize  year-end  adjustments.  Additionally,  the  policy 
identifies  the  requirement  to  accumulate  indirect  costs  both  by  expense  account  and 
organizational  department,  consistent  with  this  and  other  criteria. 

Tracking  and  auditing  for  the  allocation  and  accumulation  of  indirect  costs 
against  F-22  contracts  is  managed  by  DCMA  located  in  the  LM  Aero  facilities  in  Fort 
Worth,  Texas.  DCMA  is  responsible  for  negotiating  and  monitoring  overhead  rates  for 
all  LM  Aero  contract  efforts  based  in  Fort  Worth,  including  Spiral  2  software 
development.  Additionally,  they  received  and  monitored  all  EVMS  reports  for  the  Spiral 
2  effort,  and  identified  no  negative  findings  or  inconsistencies  with  how  LM  Aero 
allocated,  accumulated  or  reported  indirect  costs  on  the  Spiral  2  contracts. 

e.  EVM  Criterion  #20:  Identify  unit  costs,  equivalent  unit  costs,  or 
lot  costs  when  needed. 

The  focus  of  this  criterion  is  the  establishment  of  unit,  lot,  and  recurring 
costs  for  use  in  future  efforts.  In  order  to  accomplish  this,  the  contractor  must  be  able  to 
distinguish,  in  cost  accounts,  the  differences  between  recurring  (e.g.,  production)  and 
non-recurring  (e.g.,  development)  activities. 

LM  Aero  cost  accounting  does  not  support  the  tracking  of  direct  unit  or  lot 
costs.  The  method  used  to  obtain  these  values  is  an  annual  calculation  of  average  unit 
cost  for  the  specified  WBS  elements.  Segregation  of  recurring  and  non-recurring,  LM 
Aero  establishes  discreet  work  accounts  that  are  defined  based  on  recurring  or  non¬ 
recurring  activities.  The  charges  associated  with  these  efforts  can  be  summarized 
through  the  WBS  as  with  other  work  activities. 

Spiral  2  software  development  for  the  F-22  was  almost  exclusively  a  non¬ 
recurring  effort.  All  “production”  of  Spiral  2  software  was  non-recurring  and  the 
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installation  of  the  finished  software  to  aireraft  was  aecomplished  via  separate  produetion 
eontraets  (or  modifieations).  While  there  were  opportunities  to  apply  lessons  learned  to 
follow-on  F-22  development  activities,  these  could  not  be  considered  recurring  as  they 
involved  different  requirements,  schedules  and  teams.  Based  on  the  lack  of  applicability 
to  Spiral  2,  this  criterion  was  assessed  as  inconclusive. 

f.  EVM  Criterion  #21:  For  EVMS,  the  material  accounting  system 
will  provide  for: 

1.  Accurate  cost  accumulation  and  assignment  of  costs  to 
control  accounts  in  a  manner  consistent  with  the  budgets  using 
recognized,  acceptable,  costing  techniques. 

2.  Cost  performance  measurement  at  the  point  in  time  most 
suitable  for  the  category  of  material  involved,  but  no  earlier  than 
the  time  of progress  payments  or  actual  receipt  of  material. 

3.  Full  accountability  of  all  material  purchased  for  the 
program  including  the  residual  inventory. 

This  criterion  ensures  useful  measurements  of  cost  and  schedule  variance 
(CV  and  SV)  related  to  the  material  accounting  system.  It  requires  allocation  of  all 
appropriate  purchases  to  the  same  accounting  period,  thus  reflecting  planned  versus 
earned  value  (proper  recording  enables  an  accurate  SV).  This  same  expected  allocation 
process  also  aids  in  the  proper  accounting  of  earned  versus  actual  costs  (proper  recording 
enables  an  accurate  CV). 

LM  Aero  provided  a  detailed  (but  general)  description  of  the  company’s 
in-place  processes  that  deal  directly  with  this  criterion.  LM  Aero’s  Accounting  division 
established  routines  that  ensured  the  validity  of  the  data  input  (used  for  tracking  cost 
accumulation)  while  enabling  any  necessary  editing  for  “transactional  existence  and 
compatibility”  (Lockheed  Martin  Aeronautics  Company,  2005).  These  same  routines 
also  helped  maintain  appropriate  records:  specifically,  those  records  dealing  with 
requirements,  commitments,  receipts,  issues,  and  inventory  by  group,  part  number,  unit, 
and  actual  price.  The  accounting  records  also  enabled  identification  of  different  groups 
or  cost  types,  which  in  turn  permitted  summarization  of  costs  into  basic  categories  such 
as  raw  materials,  hardware,  equipment,  tooling  materials,  and  purchased  parts.  Finally, 
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the  accounting  records  also  allowed  for  subcontract  and  inter-company  work  transfers 
identification.  The  description  then  began  an  overview  of  direct  charge  materials 
(procurements),  work-in-process  accounting,  and  (contract)  inventories: 

•  Upon  receipt,  raw  materials,  hardware,  equipment,  tooling  material,  and 
purchased  parts,  along  with  major  component/subsystem  procurements, 
were  charged  to  the  Spiral  2-specific  contract  work  order.  Materials- 
related  items  such  as  tooling  and  shipping  (i.e.,  other  than  manufacturing 
materials)  were  reported  against  the  gaining  control  accounts  at  issue  from 
inventory. 

•  Sub-accounts  identified  as  work-in-process  (WIP)  provided  the  status  of  a 
given  item  in  the  process  flow  (e.g.,  on  dock,  in  inventory,  or  placed  into 
production).  WIP-related  progress  payments  were  segregated  into 
separate  WIP-inventory  accounts — under  the  buying  contract — for  unique 
cost  identification.  Upon  receipt  of  the  procured  item,  the  subcontractor's 
progress  payments  were  liquidated  and  the  value  was  recorded  to  the 
appropriate  WIP  account. 

•  Contract  inventories  were  carried  at  purchase  order  price.  Source 
documents  from  these  contract  inventories  were  utilized  to  collect  charges 
for  input  to  the  cost  accumulation  system  (from  receiving  reports, 
invoices,  requisitions,  etc).  From  that  point,  costs  were  accrued  for 
unbilled  received  items  and  unmatched  invoice  suspense  items,  such  as 
those  related  to  subsystem  vendors  and  subcontractors.  These  accruals 
were  distributed  to  the  appropriate  contract  work  orders  and  work-in- 
process  sub-accounts  each  month. 

In  practice,  incurred  or  accrued  costs  for  direct  charge  materials  and  major 
components/subsystems  procurement  were  reported  against  the  benefiting  control 
accounts  upon  issue  from  inventory  for  performance  measurement  purposes.  The  study 
failed  to  acquire  information  regarding  LM  Aero’s  material  accounting  system  as  it 
pertained  directly  to  Spiral  2.  LM  Aero  policy  meets  the  guidelines  of  this  criterion.  The 
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apparent  lack  of  visibility,  however,  into  the  Spiral  2-specific  material  accounting  system 
coupled  with  LM  Aero’s  vague,  unbilled  items  policies  and  unmatched  invoices  made  an 
unchallenged  acceptance  of  criterion  satisfaction  with  respect  to  Spiral  2  difficult.  With 
respect  to  Spiral  2  existence  within  LM  Aero’s  material  accounting  system,  the  EVMS 
program  warranted  an  inconclusive  rating  due  to  lack  of  Spiral  2-specific  supporting  data. 


4,  Group  4:  Analysis  Criteria 

Table  5  lists  the  assessments  determined  for  each  of  the  six  ANSLEIA  EVMS 
Analysis  Criteria. 


Criteria 

Assessment 

22  Identify  schedule  and  cost  variances 

Satisfactory 

23  Analyze  schedule  and  cost  variances 

Satisfactory 

24  Analyze  indirect  costs 

Satisfactory 

25  Summarize  data  elements  and  variances  for  reporting 

Satisfactory 

26  Implement  managerial  actions 

Insufficient 

27  Develop  revised  estimates  of  cost  at  completion 

Marginal 

Table  5.  Assessment  of  E-22  Spiral  2  Implementation  of  ANSI/EIA  EVMS  Analysis 


Criteria 


a.  EVM  Criterion  #22:  At  least  on  a  monthly  basis,  generate  the 
following  information  at  the  control  account  and  other  levels,  as 
necessary  for  management  control,  using  actual  cost  data  from, 
or  reconcilable  with,  the  accounting  system: 

1.  Comparison  of  the  amount  of  planned  budget  and  the 
amount  of  budget  earned  for  work  accomplished.  This 
comparison  provides  the  SV. 

2.  Comparison  of  the  amount  of  budget  earned  and  the 
actual  (applied  where  appropriate)  direct  costs  for  the  same  work. 
This  comparison  provides  the  CV. 

This  criterion  forms  the  foundation  of  EVM  reporting.  Its  focus  is  to 
compare  performance  at  the  control  account  level  with  earned  value  results.  Compliance 
with  this  criterion  should  translate  into  program  managers  identifying  potential  overruns 
and  underruns. 
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LM  Aero  stated  that  BCWS,  BCWP  and  ACWP  were  identified  for  eaeh 
eontrol  aeeount  monthly.  The  Aeeounting  Department’s  Cost  Ledger  provided  the 
ACWP  for  each  control  account.  BCWS  values  were  generated  from  work  packages 
according  to  the  PMB,  and  summarized  to  the  control  accounts  for  each  respective  cost 
element.  Budgetary  values  for  cost  elements  reported  as  earned  (for  completed  work 
packages)  and  completed  portions  of  in-process  work  packages  resulted  in  the  BCWP. 
For  work  packages  that  utilized  other-than-cost  work  measurement  systems,  target  values 
(e.g.,  standard  hours)  assigned  to  activities  were  earned  as  activities  completed. 
Comparing  the  cumulative  earned  targets  against  the  total  target  value  for  each 
performing  department,  the  work  package  percent  completion  status  is  determined  and 
used  to  calculate  BCWP  for  the  work  package.22  BCWP  for  special  cases  was  calculated 
as  follows: 

•  Work  packages  established  for  tooling  and  manufacturing  materials  and 
work  packages  established  for  procured  tools  earned  their  budgetary  dollar 
value  incrementally  as  these  materials  or  tools  were  issued  for  processing 
and/or  use. 

•  The  cost  of  any  subcontracted  items/systems  issued  was  recorded  to 
specific  accounting  WIP  sub-accounts  by  work  order.  Cumulative  actual 
costs  against  these  sub-accounts  were  compared  to  the  total  estimate  for 
these  subcontracted  items/systems  to  determine  a  percent  completion  then 
used  in  calculating  a  subcontractor  BCWP.  Budgets  for  the  nonrecurring 
effort  of  major  subcontractors  were  time -phased  within  work  packages 
according  to  the  planned  receipt  and  payment  of  each  vendor's  invoice: 
when  the  invoice  was  allocated  to  WIP,  earned  value  (BCWP)  was 
awarded. 

•  Major  subcontractors  classified  as  critical  subcontractors  (e.g.,  Boeing, 
Northrop-Grumman)  and  under  contract  for  other  than  a  firm  fixed  price 

22  For  many  discretely  measured  work  packages,  the  timing  and  amount  of  the 
budgetary  value  earned  depends  upon  activity  completion.  As  an  activity  completes, 
BCWP  was  earned  for  the  work  package  in  the  proportion  of  the  activity's  resource 
estimate  (relative  to  the  total  resource  estimate  of  the  work  package). 
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were  eontraetually  obligated  to  eomply  with  additional  eontrol  and 
reporting  criteria.  For  example,  status  reporting  in  compliance  with  ANSI 
EVMS  Criteria  provided  an  additional  cost  and  schedule  performance 
measurement  tool.  This  data  became  the  source  of  any  reported 
performance  related  to  the  subcontractor. 

In  practice,  as  stated  within  the  description  of  this  criterion,  performance 
data  was  essential  at  the  control  account  level  since  it  effectively  enabled  monitoring  of 
project  performance.  For  a  project  like  Spiral  2  that  was  organized  according  to  an  IPX 
structure,  the  performance  data  provided  the  program  manager  with  a  summary  of 
progress  and  cost  performance  on  each  WBS  element  assigned  to  the  Spiral  2  team. 
Spiral  2  data  generated  at  month-end  began  reflecting  a  favorable  performance  variance, 
confirming  the  underrun  anticipated  due  to  the  over-estimated  undefinitized  contract 
much  of  Spiral  2  existed  under.  During  the  Spiral  2  Phase  B  CPR  reporting  period  (April 
2004  through  January  2005),  the  F-22  SPO  had  no  outstanding  issues  regarding  the 
figures  generated  for  BCWS,  BCWP,  ACWP,  CV,  and  SV,  and  LM  Aero  consistently 
and  reliably  reported  them  every  month  during  the  contract  duration.23  In  conclusion, 
with  respect  to  Spiral  2’s  generation  of  BCWS,  BCWP,  ACWP,  CV,  and  SV  on  at  least  a 
monthly  basis,  the  EVMS  program  warranted  a  satisfactory  rating  for  meeting  the 
purpose  of  this  criterion. 


b.  EVM  Criterion  #23:  Identify,  at  least  monthly,  the  significant 
differences  between  both  planned  and  actual  schedule 
performance  and  planned  and  actual  cost  performance,  and 
provide  the  reasons  for  the  variances  in  the  detail  needed  by 
program  management. 

This  criterion  asserts  that  whenever  either  SV  or  CV  reported  from  a  given 
CPR  exceeds  a  previously  agreed-upon  level  between  customer  and  contractor,  the 
contractor  should  analyze  associated  drivers  and  provide  a  reason  why  that  threshold  was 


23  This  criterion  does  not  consider  the  disconnect  between  undefinitized  and 
definitized  contract  costs:  rather,  it  only  looks  for  the  successful  generation  of  the  EVM 
measures  listed. 
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broken.  This  arrangement  should  also  filter  down  to  major-subeontraetors.  Additionally, 
a  plan  for  reeovery  is  eonsidered  customary  alongside  a  given  variance  analysis. 


LM  Aero  stated  that  contract  significant  variances  were  determined  by  the 
variance  reporting  conditions  negotiated  for  the  CPR.  Whenever  a  WBS  summary  level 
element's  variance  satisfied  the  conditions  negotiated  for  CPR  analysis  (i.e.,  negatively 
surpassed  the  contracted  threshold),  the  company  documented  an  analysis  of  variances 
for  those  control  accounts  principally  responsible  for  the  summary  level  variance,  and 
inserted  a  variance  package  within  that  period’s  CPR  (if  the  customer  contracted  for  it). 
Reasons  for  significant  progress  differences  from  the  plan  were  also  identified. 

The  CPRs  related  to  Spiral  2  Phase  B  reflected  adherence  to  this  criterion. 
When  Spiral  2  Phase  B  was  contracted,  the  F-22  Program  Office  identified  the  CPR 
Format  5  (Variance  Analysis)  as  necessary  for  effective  program  oversight.  Every  month 
LM  Aero  was  contractually  required  to  report  variance  analysis.  Beginning  with  the 
October  2004  CPR,  LM  Aero  reported  significantly  positive  CVs,  indicating  the  already- 
anticipated  underrun  derived  from  definitization  of  the  project  at  a  much  lower  target 
cost.  In  conclusion,  with  respect  to  Spiral  2’s  identification  of  significant  CV  and  SV  on 
at  least  a  monthly  basis,  the  EVMS  program  warranted  a  satisfactory  rating  for  meeting 
the  purpose  of  this  criterion. 


c.  EVM  Criterion  #24:  Identify  budgeted  and  applied  (or  actual) 
indirect  costs  at  the  level  and  frequency  needed  by  management 
for  effective  control,  along  with  the  reasons  for  any  significant 
variances. 

Changes  in  indirect  costs  can  be  an  important  consideration  of  the  project 
management.  An  increase  in  indirect  charges  is  driven  by  either  an  increase  in  the 
indirect  expenses  of  the  project  or  a  decrease  in  the  direct  base  over  which  the  indirect 
charges  are  applied.  This  criterion  requires  that  changes  against  the  baseline  for  indirect 
charges  be  identified  and  adverse  impacts  be  addressed. 

The  identification  of  variances  between  indirect  budget  and  actual  indirect 
charges  is  evaluated  on  a  monthly  basis  by  each  LM  Aero  department  head. 
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Additionally,  indirect  manpower  actuals  are  collected  on  a  weekly  basis  and  variances  are 
reported  to  upper  management. 

Although  the  F-22  Program  Office  was  never  able  to  obtain  detailed 
indirect  charges  associated  with  modernization  contracts,  DCMA  who  is  responsible  for 
monitoring  all  indirect  charges  per  negotiated  rates  never  identified  any  finding  or 
concerns  for  the  Spiral  2  delivery  orders.  Based  on  this  absence  of  negative  findings  an 
assessment  of  satisfactory  was  provided  for  meeting  the  purpose  of  this  criterion. 

d.  EVM  Criterion  #25:  Summarize  the  data  elements  and 
associated  variances  through  the  program  organization  and/or 
WBS  to  support  management  needs  and  any  customer  reporting 
specified  in  the  contract. 

This  criterion  acknowledges  control  account-level  variances  are  not 
reported  simply  because  they  either  offset  themselves  (a  negative  and  a  positive)  and/or 
the  contractor  can  (and  prefers  to)  handle  such  management  details  in-house.  However, 
any  project  must  have  flexibility  in  reporting  variances.  Furthermore,  internal  and 
external  reports  must  align. 

LM  Aero  stated  that  performance  data  was  summarized  from  the  control 
accounts  through  the  WBS.  Also,  data  from  one  control  account  was  allocated  only  to  its 
unique  summary-level  WBS  element.  BCWS,  BCWP,  ACWP,  SV,  and  CV  were 
summarized  directly  to  the  reporting  level  specified  within  the  contract  in  question. 

Once  again,  the  CPRs  related  to  Spiral  2  Phase  B  reflected  adherence  to 
this  criterion.  When  Spiral  2  Phase  B  was  contracted,  the  SPO-side  of  the  F-22  program 
identified  the  CPR  Formats  1  (WBS),  2  (IPX  Structure),  and  5  (Variance  Analysis)  as 
necessary  for  effective  program  oversight  through  the  Spiral  2  Phase  B-associated  CDRL. 
Thus,  every  month  LM  Aero  was  contractually  required  to  report  data  elements  and 
variance  analysis  within  the  parameters  of  these  formats.  In  conclusion,  with  respect  to 
Spiral  2  summarization  of  data  elements  (and  associated  variances)  through  the  program 
organization  and/or  WBS  that  supported  management  and  customer  needs,  the  EVMS 
program  warranted  a  satisfactory  rating  for  meeting  the  purpose  of  this  criterion. 
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e.  EVM  Criterion  #26:  Implement  managerial  actions  taken  as  the 
result  of  EV  information. 

The  intent  of  this  eriterion  is  to  ensure  specifie  procedures  and  policies  are 
set  in  place  to  ensure  management  identifies  corrective  actions  whenever  EV  variances 
indicate  either  poor  performance  (i.e.,  negative  variances)  or  a  faulty  baseline  plan  (i.e., 
positive  variances).  Thresholds  must  be  identified  in  advance  to  trigger  management 
involvement.  These  thresholds  must  be  meaningful,  should  be  at  multiple  monitoring 
points,  and  be  in  terms  of  both  positive  and  negative  variances. 

LM  Aero  considers  the  requirement  of  this  criterion  to  be  synonymous 
with  Criterion  #22,  generation  of  monthly  reports  identifying  schedule  and  cost 
variances.  The  identification  of  predefined  variance  thresholds  are  negotiated  for  each 
contract  LM  Aero  enters  with  the  government.  According  to  LM  Aero  procedures,  once 
these  thresholds  are  breached,  actions  must  be  taken  to  both  identify  the  reason  for 
significant  variances  and  identify  the  managerial  corrective  action. 

When  looking  at  Spiral  2  DO  0002,  the  only  stand-alone  Spiral  2  contract 
complete  to  date,  compliance  with  this  criterion  was  suspect.  Although  variance 
thresholds  were  identified  (monthly:  $1M  and  +/-  10%;  cumulative:  $2M  and  +/-  10%) 
there  was  no  evidence  that  these  thresholds  meant  anything.  Throughout  the  entire  effort 
DO  0002  showed  significant  positive  cost  variances.  Live  of  the  ten  months  of  EV 
reporting  had  cost  variances  greater  than  the  $1M  threshold  set  by  the  contract.  This 
indicated  a  questionable  contract  baseline.  Although  various  root  causes  were  discussed 
for  the  significant  variances,  no  steps  were  taken  to  rebaseline  the  program  to  a  more 
realistic  plan.  At  its  completion,  DO-0002  completed  approximately  40  percent  under  its 
UCA  value,  clearly  indicating  the  PMB  managed  to  during  the  execution  of  the  contract 
was  not  an  accurate  reflection  of  contract  costs.  In  conclusion,  the  assessment  for  the  L- 
22  EVMS  implementation  of  the  criterion  associated  with  taking  necessary  managerial 
actions  was  insufficient. 
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f.  EVM  Criterion  #27  -  Develop  revised  estimates  of  cost  at 
completion  based  on  performance  to  date,  commitment  values  for 
material,  and  estimates  for  future  conditions.  Compare  this 
information  with  the  PMB  to  identify  variances  at  completion 
important  to  company  management  and  any  applicable 
customer-reporting  requirements,  including  statements  of 
funding  requirements. 

This  criterion  deals  with  the  EVM  summary  of  data  elements  known  as 
EAC.24  Routine  ealculation  of  EAC  must  ensure  both  aeouracy  and  timeliness,  as 
transgression  of  either  brings  the  entire  eost  of  the  projeet  into  uneertainty.  The  final  step 
is  to  routinely  eompare  the  EAC  with  the  PMB  to  aseertain  the  progress,  or  laek  thereof, 
of  the  program  in  question. 

EM  Aero  poliey  required  performing  comprehensive  updates  of  eost-to- 
complete  at  least  twiee  a  year,  and  more  frequently  if  direeted.  The  initial  step  in 
preparing  an  estimate  of  eost-at-eompletion  was  ensuring  all  authorized  tasks  were 
aligned  with  both  their  appropriate  WBS  element(s)  and  their  respeetive  departments 
(those  expeeted  to  perform  the  tasks).  Estimated  eompletion  dates  were  reviewed  and 
revised  as  appropriate,  with  eonsideration  given  to  performanee  to  date,  and  authorized 
tasks  not  yet  defined  and  planned  as  speeific  aetivities  were  forecast  within  undefined 
aggregates.  Notable  within  eost-to-eomplete  foreeasts  (due  to  management  and  customer 
interest)  were  direet  labor  and  overhead  rates.  The  direet  labor  rates  included  projections 
made  from  extrapolations  of  rates-to-date,  labor  union  agreements,  eompany  merit 
assumptions,  ehanges  projeeted  in  level  of  employment,  and  skill  mixes  required  to 
eomplete  the  remaining  work.  Cost-to-eomplete  overhead  rates  were  expressed  as 
applied  overhead  rates  and  forward  prieing  rates.  They  were  normally  revised  annually 
(or  more  frequently)  based  on  aetual  and  projeeted  business  eonditions.  Onee  initial 
ealeulation  of  eost-to-eomplete  was  finished,  the  project  team  determined  the  project's 
cost-to-complete  iteratively  via  proeess  of  estimation,  review,  feedback,  and  revision. 


24  See  diseussion  within  the  EVM  Primer  seetion  of  Chapter  I. 


71 


The  CPRs  related  to  Spiral  2  Phase  B  refleeted  adherence  to  the 
development  of  revised  estimates  portion  of  this  criterion.  CPR  Formats  1  and  2 
highlighted  Latest  Revised  Estimates  (LREs)  as  the  project  went  on,  showing  a  slight 
positive  CV  until  the  final  report  (January  2005)  formally  acknowledged  the  definitized 
contract  price  via  the  Management  EAC  block  in  Eormat  1.  It  is  assumed  this  late 
acknowledgement  of  such  a  significant  variance  was  forced  due  to  attempted  adherence 
to  a  somewhat-conflicting  Criterion  #30.  The  revised  estimates  provided  little  to  no 
management  control  given  the  magnitude  of  the  contract  price  change,  and  thus  at  least 
partly  failed  to  meet  the  comparison  portion  of  this  criterion.  The  fact  that  useful,  formal 
information  was  not  reported  until  the  final  CPR  validated  that  statement.  In  conclusion, 
with  respect  to  Spiral  2  development  of  revised  estimates  of  cost  at  completion  based  on 
performance  to  date,  commitment  values  for  material,  and  estimates  for  future  conditions, 
the  EVMS  program  warranted  a  marginal  rating  for  exhibiting  some  material  failures  in 
meeting  the  purpose  of  this  criterion. 

5,  Revisions  Criteria 

Table  6  identifies  the  assessment  for  each  of  the  five  criteria  under  the  ANSI/EIA 
EVMS  Revisions  Criteria. 


Criteria 

Assessment 

28  Incorporate  changes  into  plans,  budgets  and  schedules 

Satisfactory 

29  Reconcile  budgets  changes 

Inconclusive 

30  Control  retroactive  changes 

Excellent 

3 1  Control  revisions  to  the  program  budget 

Excellent 

32  Document  changes  to  the  PMB 

Satisfactory 

Table  6.  Assessment  of  E-22  Spiral  2  Implementation  of  ANSI/EIA  EVMS  Revisions 


Criteria 
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a.  EVM  Criterion  #28:  Incorporate  authorized  changes  in  a  timely 
manner,  recording  the  effects  of  such  changes  in  budgets  and 
schedules.  In  the  directed  effort  prior  to  negotiation  of  change, 
base  such  revisions  on  the  amount  estimated  and  budgeted  to  the 
program  organizations. 

Although  “timely”  is  not  defined,  it  is  reasonable  to  expeet  any  eontraetor 
and  government  team  should  be  able  to  agree  on  what  eonstitutes  a  “timely”  update  to 
project  plans  to  incorporate  necessary  changes.  Changes  may  be  driven  either  internally 
(e.g.,  significant  cost  or  schedule  overruns)  or  externally  (e.g.,  changes  in  contract  scope 
or  available  funding).  Regardless  of  the  cause,  the  owners  of  the  PMB  must  be  able  to 
update  the  plan  and  incorporate  the  necessary  changes  in  a  short  enough  time  to  minimize 
lack  of  useful  EV  data. 

The  LM  Aero  policy  governing  this  criterion  stipulates  the  contract  budget 
base  will  be  updated  following  contractual  authorization.  Although  no  specific  timeline 
can  be  provided  for  the  various  contract  modifications  that  could  occur,  the  policy  states 
that  “generally  within  60  days  after  contractual  authorization  the  change  will  be 
incorporated  into  program  schedules  and  the  performance  measurement  baseline” 
(Lockheed  Martin  Aeronautics  Corporation,  2005).  Although  the  policy  addresses 
changes  driven  by  contract  (external)  changes,  there  is  no  discussion  or  direction 
regarding  the  need  to  rebaseline  the  PMB  due  to  internal  project  execution  issues.  This  is 
addressed  in  Criterion  #32. 

Rebaselining  of  schedules  and  PMB  has  been  a  challenge  for  the  F-22 
since  the  earliest  days  of  the  program.  Based  on  F-22  records,  the  entire  development 
program  underwent  annual  rebaselining  the  first  three  years  of  the  program.  This  set  the 
tone  for  the  remainder  of  the  EMD  phase  of  the  program  and  continued  into  the  post- 
EMD  modernization  development  efforts  such  as  Spiral  2.  According  to  interviews  with 
Spiral  2  managers,  the  approach  regarding  Spiral  2  was  to  use  “rolling  baselines”  to 
accommodate  constantly  changing  schedules,  requirements  and  external  perturbations. 
While  these  frequent  changes  to  the  baseline  may  be  occurring  in  a  “timely”  manner,  the 
intent  of  this  criterion  is  not  to  merely  update  the  PMB.  It  is  to  accurately  and 
realistically  update  the  PMB  so  that  frequent  updates  of  the  program  plan  are  not 
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required.  The  ehallenge  for  the  Spiral  2  management  team  was  diseerning  when  PMB 
rebaselines  were  being  driven  by  external  faetors  versus  poor  performanee  against  the 
baseline.  The  eoneerns  assoeiated  with  this  eriterion  were  not  driven  by  the  Spiral  2 
implementation  of  the  eriterion  as  mueh  as  with  the  faetors  that  eontributed  to  the 
frequent  rebaselining;  therefore,  a  rating  of  satisfactory  was  warranted  for  this  eriterion. 

b.  EVM  Criterion  #29:  Reconcile  current  budgets  to  prior  budgets, 
in  terms  of  changes  to  the  authorized  work  and  internal 
replanning  in  the  detail  needed  by  management  for  effective 
control. 

This  eriterion  highlights  the  importanee  of  traeeability  with  respeet  to 
WBS,  speoifieally  ehanges  affeeting  the  baseline.  Program  teams  need  to  provide  this 
traeeability  to  the  lowest  level,  given  that  baselines  are  generally  developed  with  a 
bottoms-up  method.  That  traeeability  provides  the  means  for  reeoneiliation  between 
eurrent  and  prior  budgets. 

LM  Aero  asserts  that  any  of  their  programs’  eontraet  budget  bases  and/or 
PMBs  will  ehange  only  as  a  result  of  negotiations  that  result  in  eontraetual  ehange 
authorizations  or  revisions  to  proposal  values.  (PMBs  ean  also  ehange  as  a  result  of 
approved  internal  replanning).  LM  Aero  identifies  eaeh  eontraetual  ehange  and 
reeoneiles  it  to  the  original  eontraet  budget  base  and/or  PMB.  This  ensures  target  eost 
integrity  reporting  both  internally  and  to  its  eustomers. 

The  short  duration  of  Spiral  2  DO  0002  appeared  to  negate  any 
opportunity  that  might  have  provided  proof  of  adherenee  to  this  eriterion.  By  January 
2005,  when  the  ehange  to  the  total  eontraet  budget  was  formally  reeognized,  $22M  of  the 
revised  eontraet  total  of  $24M  had  already  been  aeeomplished.  Thus,  per  the  Spiral  2  DO 
0002  CDRL,  with  over  95  pereent  of  the  work  aeeomplished,  CPRs  were  no  longer 
required.25  The  study  failed  to  aequire  information  regarding  LM  Aero’s  traeeability  of 
budget  ehanges  as  it  pertained  direetly  to  Spiral  2.  Though  LM  Aero  poliey  met  the 


25  $22M  was  95  pereent  of  the  non-award  fee  total  of  the  definitized  eontraet  budget 
base. 
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guidelines  of  this  eriterion,  the  apparent  laek  of  visibility  into  Spiral  2-speoilio  budgetary 
ehanges  made  an  unehallenged  aeeeptanee  of  eriterion  satisfaetion  with  respeet  to  Spiral 
2  alone  diffieult.  As  such,  with  respect  to  Spiral  2’s  ability  to  reconcile  budgetary 
changes,  the  EVMS  program  warranted  an  inconclusive  rating  due  to  lack  of  Spiral  2- 
specific  supporting  data. 


c.  EVM  Criterion  #30:  Control  retroactive  changes  to  records 
pertaining  to  work  performed  that  would  change  previously 
reported  amounts  for  actual  costs,  EV,  or  budgets.  Adjustments 
should  be  made  only  for  correction  of  errors,  routine  accounting 
adjustments,  effects  of  customer-  or  management-directed 
changes,  or  to  improve  the  baseline  integrity  and  accuracy  of 
performance  measurement  data. 

The  intent  of  this  criterion  is  to  ensure  the  integrity  of  the  EVMS.  If 
budgets  or  actuals  are  changed  after-the-fact,  the  usefulness  of  the  EVMS  as  a 
management  control  tool  is  lost.  The  only  appropriate  retroactive  updates  are  those  made 
to  correct  errors  or  other  legitimate  accounting  adjustments. 

It  was  not  surprising  to  find  EM  Aero  policy  specifically  prohibits  changes 
to  previously  reported  actual  costs  unless  it  is  done  as  a  correction  of  errors  or  an 
accounting  adjustment.  Even  then,  the  policy  stipulates  any  changes  to  previously 
reported  EV  data  must  be  coordinated  through  and  approved  by  senior  management. 

Inappropriate  changes  to  reported  data  would  be  a  serious  infraction  of 
contractual  obligations  and  a  breach  of  trust.  There  has  never  been  an  indication  through 
audits  or  otherwise  that  EM  Aero  would  jeopardize  their  relationship  with  the 
government  or  their  EVMS  certification  to  retroactively  change  EV  data.  As  such  an 
excellent  rating  was  warranted  for  this  criterion. 


d.  EVM  Criterion  #31  -  Prevent  revisions  to  the  program  budget, 
except  for  authorized  changes. 

Virtually  all  DoD  programs  experience  challenges  to  the  plan  originally 
put  forward  at  the  beginning  of  the  effort.  Sometimes  these  challenges  become  so  severe 
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that  project  managers  need  to  change  the  budget  baseline  to  a  more  realistic  budget 
baseline.  This  criterion  recognizes  baselines  may  change,  but  any  changes  to  the  budget 
associated  with  the  baseline  must  be  changed  on  a  limited  basis  and  in  a  controlled 
environment. 

In  the  case  of  LM  Aero,  EVMS  policy  states  that  a  contract  budget  base 
can  only  be  changed  when  authorized  through  a  proposal  update.  This  policy  ensures  the 
contractor  will  never  unilaterally  change  the  budget  baseline.  Based  on  this  strong  policy 
and  observed  performance  in  this  area,  an  excellent  rating  was  warranted  for  this 
criterion. 


e.  EVM  Criterion  #32:  Document  changes  to  the  PMB. 

As  changes  occur  to  the  project  PMB,  these  changes  must  be  controlled 
and  traceable.  Unauthorized  changes  to  the  PMB  undermine  the  utility  of  the  EVMS. 
Similarly,  when  an  update  to  the  baseline  cannot  be  traced  to  the  original  plan,  it  becomes 
difficult  to  identify  where  trade-offs  occurred,  impacts  to  other  areas  of  the  program,  or 
simply  program  history. 

Under  Criterion  #10,  EM  Aero  typical  policy  is  to  define  detailed  work 
packages  for  all  efforts  within  the  current  contract  year.  Budgets  for  activities  planned 
beyond  the  current  year,  particularly  those  in  support  of  development  efforts,  are  held  in 
planning  packages  until  more  is  known  regarding  the  activity’s  details.  As  the  project 
progresses  and  these  planning  packages  are  transitioned  into  detailed  work  packages, 
these  changes  are  captured  in  documentation,  showing  the  relationships  between  the 
planning  packages  and  work  packages.  Additionally,  changes  driven  by  external 
program  pressures,  as  discussed  in  Criterion  #28,  are  to  be  documented  and  identify 
linkages  to  the  previous  PMB. 

As  discussed  in  Criterion  #28,  the  Spiral  2  PMB  was  updated  using 
“rolling  baselines”  to  accommodate  for  external  perturbations.  Unfortunately,  while 
these  changing  baselines  showed  linkages  to  previous  PMB,  where  possible,  many  times 
the  changes  to  the  PMB  were  significant  enough  that  traceability  between  activities 
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became  very  difficult.  Additionally,  it  was  difficult  to  identify  which  changes  occurred 
in  the  PMB  as  a  result  of  external  pressures  versus  poor  internal  performance.  The 
inability  of  project  managers  to  identify  these  differences  limits  the  utility  of  the  EV  data. 
These  concerns  notwithstanding,  the  F-22  documentation  of  changes  to  the  PMB  was 
assessed  to  be  sufficiently  in  keeping  with  the  criterion,  warranting  a  satisfactory  rating. 

C.  SUMMARY 

Table  7  summarizes  the  authors’  assessments  of  how  each  of  the  32  ANSI/EVMS 
criteria  was  applied  on  E-22  Spiral  2.  While  there  were  several  criteria  not  met  (or  which 
lacked  the  requisite  verification),  there  was  not  a  systemic  failure  in  the  EVMS  process  at 
E-22.  Rather,  most  of  the  challenges  being  faced  by  the  use  of  EVMS  on  Spiral  2  were 
driven  by  issues  outside  of  the  EVMS  spectrum.  This  will  be  further  discussed  in 
Chapter  IV,  F-22  EVMS  Environment,  where  the  F-22  development  environment  may 
have  contributed  to  difficulties  in  EVMS  implementation. 


77 


Criteria 

Assessment 

Group  1:  Organization 

1 

Define  authorized  work  (WBS  elements) 

Satisfactory 

2 

Identify  organizational  responsibilities 

Satisfactory 

3 

Integrate  the  system 

Satisfactory 

4 

Identify  overhead  management 

Excellent 

5 

Provide  for  performanee  measurement 

Satisfactory 

Group  2:  Planning,  Scheduling  and  Budgeting 

6 

Schedule  the  work 

Marginal 

7 

Identify  products,  milestones  and  indicators 

Satisfactory 

8 

Plan  the  Performance  Measurement  Baseline  (PMB) 

Marginal 

9 

Establish  budgets  for  work 

Inconclusive 

10 

Identify  work  packages 

Satisfactory 

11 

Summarize  work  package  budgets  to  control  accounts 

Satisfactory 

12 

Identify  and  control  level  of  effort 

Marginal 

13 

Establish  overhead  budgets 

Satisfactory 

14 

Identify  management  reserves  and  undistributed  budget. 

Excellent 

15 

Summarize  budgets  to  target  cost 

Excellent 

Group  3:  Accounting 

16 

Record  direct  costs 

Excellent 

17 

Summarize  direct  cost  to  the  WBS 

Satisfactory 

18 

Summarize  direct  cost  to  the  organization 

Marginal 

19 

Record  indirect  costs 

Excellent 

20 

Identify  unit/lot  costs 

Inconclusive 

21 

Record  material  costs 

Inconclusive 

Group  4:  Analysis 

22 

Identify  schedule  and  cost  variances 

Satisfactory 

23 

Analyze  schedule  and  cost  variances 

Satisfactory 

24 

Analyze  indirect  costs 

Satisfactory 

25 

Summarize  data  elements  and  variances  for  reporting 

Satisfactory 

26 

Implement  managerial  actions 

Insufficient 

27 

Develop  revised  estimates  of  cost  at  completion 

Marginal 

Group  5:  Revisions 

28 

Incorporate  changes  into  plans,  budgets  and  schedules 

Satisfactory 

29 

Reconcile  budgets  changes 

Inconclusive 

30 

Control  retroactive  changes 

Excellent 

31 

Control  revisions  to  the  program  budget 

Excellent 

32 

Document  changes  to  the  PMB 

Satisfactory 

able  7.  Assessment  of  F-22  Spiral  2  Implementation  of  ANSI/EIA  EVMS  Criteria 
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IV.  F-22  EVMS  ENVIRONMENT 


A.  PREFACE 

While  the  previous  ehapter  sought  to  answer  if  the  F-22  EVMS  was  implemented 
per  industry  standards  established  for  specifically  that  purpose,  this  chapter  examines  if 
the  F-22  EVMS  fulfills  its  role  as  a  management  control  system  for  the  avionics  software 
development  program  in  question  and  if  F-22  managers  used  EVM  products  to  manage 
the  avionics  development  effort.  In  answering  each  of  these  questions,  however,  it  was 
important  to  consider  the  environment  within  which  the  F-22  EVMS  was  employed. 
During  the  course  of  gathering  data,  collecting  questionnaire  responses,  and  discussing 
the  use  of  EVM  with  F-22  managers,  the  authors  identified  certain  foundational  elements 
necessary  for  successful  implementation  of  EVM  not  addressed  by  the  32  ANSI/EIA 
EVMS  criteria.  These  are  foundational  issues  behind  any  successful  development 
program  regardless  of  EVM  use;  however,  many  of  these  are  often  cited  as  the  challenges 
facing  today’s  DoD  software  development  efforts.  Examples  include  requirements 
stability,  schedule  stability,  funding  stability  and  a  realistic  PMB.  In  the  first  section  of 
this  chapter  three  limiting  factors  or  barriers  were  examined  for  their  impact  on  the 
success  of  EVMS  implementation  in  the  F-22  Spiral  2  program.  Next,  the  question  “To 
what  degree  did  the  F-22  Spiral  2  implementation  of  EVM  fulfill  its  role  as  a 
management  control  system  for  avionics  software  development?”  was  addressed  by 
looking  at  CPRs  for  the  avionics  development  program.  Finally,  if  EVMS  is  to  be 
valuable  to  managers,  they  must  have  confidence  in  the  tool  and  be  knowledgeable  about 
EVM  and  the  data  being  generated.  The  final  section  of  this  chapter  reports  the  analysis 
of  the  questionnaire  provided  to  Spiral  2  government  and  contractor  personnel  and 
intended  to  address  this  question. 

B,  EVMS  BARRIERS  IN  THE  F-22  PROGRAM 

1,  Undefinitized  Contracts 

As  discussed  in  Chapter  II,  Literature  Review,  at  the  end  of  EMD  the  F-22 
program  recognized  the  challenges  faced  during  EMD  needed  to  be  addressed  in  the 
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modernization  program  to  prevent  a  repeat  of  eost  overruns,  sehedule  delays  and 
undelivered  performanee.  One  of  the  approaehes  implemented  to  address  the  fluid 
requirements,  eonstantly  ehanging  funding  levels  and  teehnologieal  ehallenges  was  the 
method  of  eontraeting  for  eaeh  ineremental  (“Spiral”)  produet  delivery  via  phased 
eontraets. 

Figure  8,  Spiral  2  Summary  Sehedule,  shows  how  eaeh  of  these  eontraets  was 
phased  with  respeet  to  eaeh  other  during  the  development  of  Spiral  2.  The  approaeh  of 
having  multiple  eontraet  phases  for  eaeh  spiral  provided  the  desired  programmatie 
flexibility;  however,  there  was  a  eost  to  the  business  side  of  the  program.  With  more 
eontraets  in  work  than  ever  before  and  the  remaining  issue  of  eonsistent  ehanges  in 
program  baselines,  the  award  of  negotiated,  definitized  eontraets  beeame  unattainable. 
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Figure  8.  Spiral  2  Summary  Sehedule 

(From;  DO  0019  Integrated  Baseline  Review) 


By  the  time  Spiral  2  was  initiated  in  2003,  the  F-22  program  frequently  used 
UCAs  in  initiating  new  efforts  with  LM  Aero.  The  eomplexities  of  the  modernization 
program,  with  multiple  inerements  at  various  stages  all  dealing  with  funding 
perturbations  and  requirement/teehnology  iterations,  made  modernization  even  more 
suseeptible  to  UCAs.  Not  an  anomaly  as  mueh  as  the  norm.  Spiral  2  had  all  three  of  its 
development  eontraets  initiated  under  UCAs.  In  faet,  LM  Aero  eompleted  the  first  two 
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phases  of  the  Spiral  2  program  before  the  respeetive  eontracts  were  definitized. 
Considering  the  undefinitized  nature  of  UCAs,27  the  use  of  EVM  was  extremely  diffieult, 
if  not  impossible. 

As  diseussed  in  the  previous  chapter,  the  second  group  of  criteria  deals  with 
planning,  scheduling  and  budgeting  the  program.  If  a  program  was  initiated  under  a 
UCA,  many  times  it  was  due  to  the  immaturity  of  the  program  plan.  Sometimes  this  was 
due  to  lack  of  understanding  of  the  effort  At  times  it  was  due  to  an  inability  to  define  this 
knowledge  in  terms  of  a  negotiated  definitive  contract.  Most  times,  as  was  the  case  with 
Spiral  2,  it  was  a  mixture  of  both. 

In  some  cases  the  scope  of  the  work  awarded  under  the  UCA  was  defined  so 
immaturely,  it  was  impractical  to  hold  an  IBR  early  enough  in  the  contract  to  capture  a 
significant  portion  of  the  work  to  be  performed.  DO  0019  was  an  example  of  this  with 
work  being  initiated  on  this  effort  in  January  2005  under  a  limited  UCA  and  the  IBR 
being  accomplished  in  January  2006.  Clearly  EVMS  has  limited  value  in  instances  such 
as  these  where  an  IBR  can  not  be  held  to  establish  the  baseline.  When  one  year  of  effort 
(out  of  a  total  of  three)  was  accomplished  before  an  IBR  was  held,  it  severely  limited  the 
manager’s  ability  to  measure  progress  against  a  program  baseline. 

In  the  case  of  DO  0002,  a  program  baseline  was  established  via  an  IBR  while 
under  a  UCA;  however,  there  were  significant  limitations  to  the  value  of  PMB 
established  at  this  IBR.  These  limitations  were  driven  by  several  factors,  including: 

•  A  lack  of  confidence  in  the  final  contract  value 

•  Eack  of  buy-in  between  the  parties  on  required  tasks  to  complete  the  work 
(i.e.,  contract  scope) 


26  Recall  from  Chapter  II  the  first  phase  of  Spiral  2,  requirements  analysis,  was 
actually  accomplished  under  REDI  DO  0001,  the  overarching  modernization  program’s 
systems  engineering  and  program  management  effort.  This  DO,  initiated  in  January  2003 
via  a  UCA,  was  not  definitized  until  after  Spiral  2  scope  was  completed  in  March  2004. 
DO  0002,  detailed  design,  was  awarded  via  a  UCA  in  March  2004  and  was  also 
definitized  well  after  task  completion  in  Eebruary  2005.  DO  0019,  awarded  as  a  UCA  in 
Eebruary  2005,  was  definitized  in  October  2005. 

27  Typically,  UCA’s  are  based  on  not-to-exceed  (NTE)  values  without  much,  if  any, 
supporting  data. 
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•  The  inability  of  the  contractor  to  fully  assign  dollars  to  all  Cost  Account 
Managers  (CAM). 

Regardless  of  whether  an  IBR  was  held  and  a  program  baseline  established  none 
of  the  PMB  was  likely  to  be  contractual  under  a  UCA.  This  resulted  in  difficulty  for  the 
government  in  controlling  changes  to  the  baseline  during  the  execution  of  the  UCA,  or 
when  the  contract  was  later  definitized. 

Spiral  2  DO  0002,  detailed  design,  was  an  example  of  this  challenge.  Originally 
awarded  as  a  UCA  based  on  an  NTE,  DO  0002  IBR  was  accomplished  less  than  90  days 
after  UCA  award;  however,  with  so  much  of  the  effort  based  on  NTE  values,  it  was 
difficult  for  government  managers  to  assess  the  validity  of  the  PMB  during  the  IBR. 
Ultimately,  DO  0002  would  be  completed  under  a  UCA  and  was  definitized  after 
completion  of  work  for  42  percent  less  dollars  than  originally  contracted  under  the  UCA. 
With  this  much  of  a  difference  between  the  work  projected  and  the  work  performed, 
EVM  was  difficult  to  use  as  a  management  control  tool.  As  the  contractor  team 
constantly  ran  below  budget  for  the  effort,  government  managers  were  always  faced  with 
the  question  of  determining  how  much  of  the  underrun  was  due  to  efficient  performance 
versus  excessive  budget.28  This  lack  of  confidence  in  the  PMB  significantly  reduced  its 
value  as  a  management  tool. 


2,  Requirements  Instability 

Requirements  instability  is  one  of  the  most  commonly  cited  problems  with 
suffering  software  development  programs.  With  its  changing  mission,  the  P-22  was 
especially  susceptible  to  changing  requirements  during  the  early  stages  of  Spiral  2. 
While  any  large  DoD  weapon  system  program  will  be  challenged  with  shifting 
requirements  (called  “creep”),  the  addition  of  the  air-to-ground  capabilities  to  the  existing 


28  At  the  end  of  DO  0002,  EM  Aero  was  commended  for  aggressively  implementing 
process  improvements  that  enabled  increased  performance  and  criticized  (in  award  fee 
and  past  performance  documentation)  for  excessive  cost  estimating.  One  of  the 
responses  EM  Aero  provided  to  this  critique  was  that  the  DO  0002  excessive  underrun 
was  more  due  to  changing  requirements  than  poor  budgeting.  The  following  section 
looks  at  Spiral  2  requirements  stability. 
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air-to-air  mission  of  the  F-22  presented  especially  challenging  pressures  on  the 
modernization  program  to  incorporate  capabilities  that  had  not  been  planned  for  the 
fighter.  In  addition  to  the  changing  mission  of  the  program,  the  senior  managers  of  F-22 
were  fighting  to  retain  the  program  funds  as  DoD  sought  dollars  to  pay  for  Global  War  on 
Terror  operations.  This  resulted  in  capabilities  being  promised  to  senior  DoD  leadership 
and  congress  prior  to  any  evaluations  of  the  impact  of  incorporation  into  the  subject 
Spiral.  Although  the  Spiral  development  model  was  established  to  maximize  the 
program’s  ability  to  adjust  requirements  as  required,  it  was  never  intended  to  facilitate  the 
frequent  changes  in  requirements  that  Spiral  2  experienced. 

As  requirements  analysis  neared  completion  in  January  2004,  the  requirements 
baseline  was  established  for  detailed  design.  This  set  of  requirements  was  the  basis  for 
the  DO  0002  UCA.  In  March  2004,  shortly  after  DO  0002  was  authorized,  the  first  letter 
was  written  to  LM  Aero  changing  the  major  capabilities  being  developed  for  Spiral  2.  In 
the  summer  of  2004,  funding  constraints  and  external  political  pressures  forced  the 
program  to  formally  direct  requirement  changes  to  LM  Aero  four  times  between  July  and 
September.  This  instability  in  core  Spiral  2  requirements,  made  the  use  of  a  PMB  for  DO 
0002  almost  impossible.  Every  iteration  of  requirements  changes  led  to  a  major 
rebaseline  of  the  program  plan.  This,  coupled  with  the  cultural  effect  of  knowing 
requirement  changes  were  always  being  considered,  led  to  lack  of  confidence  in  the  PMB 
and,  therefore,  the  EVM  data  on  which  it  was  based.  Operating  in  this  environment  of 
constantly  changing  baselines  yielded  what  was  commonly  referred  to  as  “rolling 
baselines. ”29  Rolling  baselines  were  key  indicators  of  the  lack  of  program  stability. 
Whether  the  instability  was  internal  or  external,  the  impact  was  the  same:  loss  of 
confidence  in  EVM.  Although  the  EVM  process  is  designed  with  the  intent  of  absorbing 
changes  to  the  program  baseline,  when  a  program  makes  changes  to  its  baseline  as 
frequently  as  the  E-22  Spiral  2  program  did,  the  value  provided  by  EVM  begins  to 
deteriorate  as  confidence  is  lost  in  the  existing  baseline. 


29  Similar  to  (and  often  a  result  of)  requirements  creep,  this  nomenclature  describes  a 
flexible  baseline  that  lacks  the  stability  essential  for  benchmarking  performance 
measurements. 
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An  additional  problem  with  frequent  rebaselines  was  the  inereased  diffieulty  in 
identifying  the  history  of  the  PMB  updates.  While  the  last  five  of  the  32  ANSI/EIA 
EVMS  eriteria  specifically  address  the  need  to  control  changes  in  PMB  and  other  EVM 
baselines,  the  more  frequently  changes  occurred,  the  more  difficult  it  became  to  identify 
the  relationships  between  current  and  former  baselines.  As  changes  continued  to  be  made 
to  the  program  baseline,  whether  in  scope,  resources,  or  schedule,  it  became  more 
difficult  to  trace  the  relationship  between  the  original  baseline  and  its  EAC  and  the 
current  baseline  and  its  EAC.  This  was  critical  for  a  program  constantly  being  pressured 
to  justify  funding  and  provide  measures  of  remaining  effort. 

3,  Timeliness  of  EVM  Reports 

The  EVM  reporting  cycle  required  in  all  Spiral  2  contracts  was  monthly.  While 
Criteria  #22  and  #23  identify  a  minimum  EVM  reporting  cycle  of  one  month,  most  of  the 
E-22  government  users  identified  this  reporting  cycle  as  being  insufficient  to  enable  use 
of  EVM  as  a  management  tool.  EM  Aero  management  and  IPT  leads  had  access  to 
preliminary  EVM  data  on  at  least  a  weekly  basis;  however,  unless  government  managers 
established  a  trusting  relationship  with  their  EM  Aero  counterparts,  they  did  not  see  any 
of  this  data  until  the  formal  EVM  reports  were  delivered  four  to  six  weeks  after  the  work 
was  completed. 

The  primary  driver  for  this  delay  is  the  necessity  of  EM  Aero  to  complete  their 
monthly  cost  accounting  before  developing  and  delivering  reports.  If  formal  reports  were 
provided  to  the  government  prior  to  the  end  of  the  accounting  period  changes  may  have 
been  required  at  the  end  of  the  period.  While  many  in  the  government  would  likely 
understand  this  situation  and  trade  it  for  the  ability  to  see  the  data  earlier,  it  does  expose 
EM  Aero  and  the  program  leadership  to  potential  problems  if  they  make  decisions  based 
on  preliminary  data.  Additionally,  as  changes  occurred  at  the  end  of  the  accounting 
periods,  there  might  be  many  who  would  lose  confidence  in  the  accuracy  of  the 
preliminary  data. 
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Based  on  discussions  with  F-22  government  managers,  it  was  determined  the 
desired  approach  was  to  establish  a  trusting  relationship  with  the  contractor  manager  to 
gain  as  much  access  as  possible  to  preliminary  EVM  data.  This  enabled  access  to  timely, 
if  not  completely  accurate,  data  that  supported  management  making  decisions.  Still  this 
was  an  informal  agreement  and  was  completely  based  on  the  “good  will”  of  the  LM  Aero 
manager.  It  also  limited  government  managers  in  their  ability  to  report  emerging  issues 
up  their  management  chain.  The  first  reason  for  this  was  because  it  was  based  on 
preliminary  data  that  could  change.  The  second  concern  was  that  it  would  “spoil”  the 
relationship  with  the  contractor  manager  who  would  be  less  likely  to  provide  access  to 
preliminary  EVM  data. 

C.  SPIRAL  2  EVMS  DATA  ANALYSIS 

This  section  primarily  addressed  the  second  question  posed  within  Methodology, 
Chapter  I:  “To  what  degree  did  the  E-22  Spiral  2  implementation  of  EVM  fulfill  its  role 
as  a  management  control  system  for  avionics  software  development?”  To  help  answer  the 
question.  Cost  Performance  Reports  (CPRs)  directly  related  to  Spiral  2  efforts  were 
distilled  into  raw  cost  and  schedule  measurements.  Then  a  Spiral  2  DO  0002  lifecycle 
EVM  analysis  was  accomplished  and  compared  to  outputs  from  EM  Aero  reports.  The 
comparison,  in  addition  to  the  EVM  data  elements’  results  and  pertinent  testimony  from 
individuals  involved,  answered  the  question. 

In  order  to  collect  the  data  necessary,  CPRs  from  the  REDI  contract  modification 
known  as  Spiral  2  DO  0002  were  gathered,  the  complete  set  dating  from  April  2004 
through  January  2005.  Using  the  raw  numbers  from  the  Eormat  2  reports,  an  Excel 
spreadsheet  was  created  that  took  the  lowest  level  IPT  EVM  data  elements,  summed  them 
up  through  the  IPT  levels  (as  opposed  to  inputting  data  from  the  CPRs’  higher  level 
IPTs),  and  generated  trend  lines  for  EVM  data  element  analysis  (e.g.,  BCWS,  BCWP, 
ACWP,  etc.).  Please  refer  to  Appendix  I  for  the  complete  set  of  monthly  reports. 
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1.  CPR  and  Independent  Analysis  Comparison 

Beginning  with  the  first  CPR  (April  2005),  an  interesting  disconneet  appeared 
between  SV  and  CV.  SV  results  put  the  program  behind  schedule  by  $31,000,  while  CV 
exhibited  a  cost  underrun  of  $660,000 — 45  percent  of  the  BCWP  for  that  period. 
Conventional  wisdom  normally  leads  to  the  conclusion  that  behind  schedule  equates  to  a 
cost  underrun,  but  usually  at  a  somewhat  similar  measurement  (e.g.,  SV  =  10,  CV  =  12). 
A  twenty-one-fold  increase  from  SV  to  CV  indicated  either  one  of  two  conclusions,  or 
some  combination  of  both:  efficiencies  were  occurring  within  one  or  more  IPTs,  or  the 
original  budget  assumptions  were  grossly  incorrect.  Regardless,  the  EVM  data  was 
highlighting  an  area  of  concern  for  program  management. 

Subsequent  reports  only  heightened  that  concern.  By  midway  through  the  CPR 
reporting  cycle  (August  2004),  the  CV  indicated  a  $4.9M  underrun — 56  percent  of  the 
ACWP  cumulative-to-date — while  SV  indicated  a  less  unwieldy  $63,000  behind 
schedule.  That  said,  LM  Aero’s  Format  5  (Variance  Analysis)  for  that  period  stated, 
“There  [were]  no  current  period,  cumulative-to-date  or  at-completion  variances  which 
exceed[ed]  thresholds  in  [August  2005] ’s  report.”  Thus,  even  though  the  EVM  data  was 
indicating  an  area  of  concern  regarding  Spiral  2  DO  0002  performance,  EM  Aero’s 
analysis  reports  to  the  F-22  SPO  indicated  all  EVM  data  elements  were  within  tolerances. 
Not  until  the  October  2004  report  did  LM  Aero’s  formal  Variance  Analysis  begin  to  offer 
a  reason  behind  the  recurring,  outsized  CVs. 

In  a  similar  case,  this  study’s  independent  EVM  analysis  uncovered  an  area  of 
concern  regarding  the  CPR  reports’  data  summarization.  In  the  June  2004  report,  the 
independent  analysis  revealed  a  relatively  minor  anomaly  in  reporting  that  continued  into 
the  final  report  (January  2005).  Whereas  the  CPR  in  question  reported  BCWS 
cumulative-to-date  as  $6.775M  for  the  entire  program,  the  independent  analysis  reported 
the  same  data  element  as  $6.863M,  resulting  in  a  delta  of  $88,000.  Though  small  in 
relative  size,  this  anomalous  delta  effected  an  investigation  into  the  cause.  The 
investigation  found  that  when  the  May  2004  CPR’s  BCWS  cumulative-to-date 
($4.437M)  was  added  to  the  June  2004  CPR’s  BCWS  current  period  ($2.428M),  the 

result  netted  a  $6.865M  amount  that  should  have  corresponded  to  June  2004  BCWS 
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cumulative-to-date.  Note  this  figure  results  in  a  minor  $2,000  delta  between  the 
independent  analysis’  same  figures  (easily  explained  by  rounding  error).  As  already 
disclosed,  this  error  continued  through  subsequent  CPRs,  ranging  in  value  from  $82,000 
to  $134,000. 

Successive  study  netted  a  similar  anomaly.  Beginning  with  the  June  2004  CPR 
report,  the  independent  analysis  reported  the  BCWP  cumulative-to-date  data  element  as 
$6.694M;  when  compared  to  the  actual  CPR’s  corresponding  value  of  $6.562M,  this 
resulted  in  a  delta  of  $132,000.  Again,  the  investigation  found  when  the  May  2004 
CPR’s  BCWP  cumulative-to-date  ($4.570M)  was  added  to  the  June  2004  CPR’s  BCWS 
current  period  ($2.127M),  the  result  netted  a  $6.697M  amount  that  should  have 
corresponded  to  June  2004  BCWS  cumulative-to-date.  Note  that  this  figure  results  in  a 
minor  $3,000  delta  between  the  independent  analysis’  same  figures  (easily  explained  by 
rounding  error).  Like  before,  this  phenomenon  continued  throughout  the  remaining 
CPRs,  ranging  in  value  from  $1 14,000  to  $166,000. 

In  both  BCWP  and  BCWS  cases,  the  Mission  Sys  &  SW  (Avionics  Systems 
Engineering  Integration  Team  and  Mission  Avionics  Software)  and  Modernization  IPTs 
were  the  cost  drivers  for  all  affected  CPRs.  Investigation  revealed  the  “What?”  portion 
of  the  cause,  but  failed  to  uncover  the  “Why.”  These  inexplicable  errors  raise  concerns 
regarding  the  validity  and  veracity  of  the  reports  given  to  the  government. 

2,  EVM  Trend  Analysis 

This  study’s  trend  analysis  benefited  from  hindsight.  Given  the  facts  known  after 
the  program  concluded,  the  Spiral  2  DO  0002  program  was  clearly  overestimated,  a  fact 
unknown  to  management  until  the  effort  was  completed  in  early  2005.  In  theory,  the 
EVM  data  elements  should  have  reflected  that  fact.  The  following  results — acquired 
from  the  independent  EVM  analysis — exhibited  trends  that  supported,  at  least  in  part,  the 
conclusion  that  the  data  elements  did  forewarn  of  problems  related  to  program  budget 
assumptions,  which  in  turn  translated  to  a  probable  exaggeration  of  BCWS  and  BCWP. 
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A  top  line  analysis  of  current-period  EVM  data  elements  revealed  that  whatever 
the  budget  assumptions  were  that  comprised  the  Spiral  2  DO  0002  undefinitized  contract 
baseline,  the  work  continued  closely  to  plan  throughout  much  of  the  program.  The 
BCWS,  BCWP,  and  SV  lines  in  Figure  9  clearly  denoted  this.  The  figure  also  clearly 
indicated  real  costs  lagged  budgeted  for  all  but  one  period  of  the  program  via  the  ACWP 
line  (below  both  BCWS  and  BCWP)  and  resultant  CV  line  (positive  for  all  but  one 
reporting  period).  These  results  lead  to  the  conclusion  that  budgeted  costs  (represented 
by  BCWS  and  BCWP)  were  overestimated  relative  to  actual  costs  (i.e.,  ACWP). 


I  BCWS 

BCWP 

ACWP 

SV 

I  —  CV 


Figure  9.  F-22  Modernization  RFDI  D00002  Data  Elements  (Current) 


A  top  line  analysis  of  cumulative  EVM  data  elements  supported  the  preceding 
conclusion  (see  Figure  10).  Throughout  the  lifecycle  of  the  program,  BCWS  and  BCWP 
remained  similar  while  ACWP  lagged  both.  As  a  result,  SV  remained  relatively  flat  and 
CV  grew.  These  results  lead  to  the  conclusion  that  budgeted  costs  (represented  by 
BCWS  and  BCWP)  were  overestimated  relative  to  actual  costs  (i.e.,  ACWP). 


88 


-^BCWS 

—  BCWP 
ACWP 
SV 

—  CV 


Figure  10.  F-22  Modernization  REDI  D00002  Data  Elements  (Cumulative) 


The  general  question  of  whether  EVM  identified  a  disconnect  between  program 
budgeted  costs  and  actual  costs  answered,  did  EVM  identify  any  other  leading  indicators 
warranting  management’s  attention?  To  answer  that  question,  the  independent  analysis 
looked  at  each  EVM  data  element  in  terms  of  IPT  totals  (see  Eigures  11  and  12).  As 
expected,  the  totals  supported  the  suspicion  of  incorrect  budgetary  estimates.  They  also 
lead  to  the  conclusion  that  much  of  the  blame  for  the  high  CV  lay  with  Mission  Sys  & 
SW.  This  particular  IPT  contributed  the  majority  of  cost  to  every  data  element  analyzed. 
Knowing  that,  program  management,  theoretically,  could  have  focused  management 
control  efforts  on  work  occurring  within  that  IPT.  More  to  the  point,  the  data  provided 
by  EVM  indicated  program  instabilities  in  addition  to  identifying  areas  of  the  program 
which  warranted  closer  attention. 
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Figure  1 1 .  F-22  Modernization  Spiral  2  DO  0002  Data  Elements  Summed  by  IPT  (Current 

Period) 
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Figure  12.  F-22  Modernization  Spiral  2  DO  0002  Data  Elements  Summed  by  IPX 

(Cumulative-to-Date) 
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D,  QUESTIONNAIRE  ANALYSIS 

This  section  addresses  the  final  question  posed  in  this  paper,  “To  what  extent  did 
the  F-22  program  management  (Government  and  Contractor)  use  EVM  products  to 
manage  avionics  development  efforts?”  The  method  chosen  to  address  this  question  was 
the  use  of  a  questionnaire.  Following  is  an  overview  of  the  questionnaire  and  an  analysis 
of  the  responses. 

1.  Questionnaire  Overview 

The  questionnaire  (see  Appendix  II)  aimed  to  assess  the  perceived  usefulness  of 
EVM  within  a  software  development  context.  The  questionnaire  consisted  of  nine 
closed-ended  questions  falling  into  three  distinct  areas.  The  first  area  (Questions  #1  and 
#2)  focused  on  the  respondent’s  background  by  asking  them  to  categorize  themselves  by 
government  or  contractor  function(s)  and  area(s)  of  focus  within  F-22  development.  The 
second  area  (Questions  #3  and  #4)  of  the  questionnaire  sought  to  establish  the 
respondent’s  level  of  interaction  with  EVM  by  looking  for  their  frequency  and  method  of 
use.  The  final  area  (Questions  #5  through  #9)  built  on  the  respondent’s  interaction  with 
EVM  and  sought  their  assessment  of  EVM.  Questions  in  this  area  focused  on  the 
respondent’s  perceived  value  and  usefulness  of  F-22  EVMS  and  its  data  as  well  as  a  self- 
assessment  of  their  knowledge  of  EVM  and  the  32  ANSI/EIA  criteria.  Those  identifying 
themselves  as  being  aware  of  the  32  ANSI/EIA  criteria  were  asked  to  assess  the  F-22 
implementation  of  EVM  with  respect  to  these  32  criteria. 

The  questionnaire  was  provided  to  a  mix  of  26  government  and  contractor  F-22 
personnel.  They  included  program  managers,  engineers,  contracting  officers,  and 
financial  managers  who  worked  on  the  F-22  program.  All  government  and  contractor 
personnel  directly  managing  Spiral  2  development  were  provided  the  questionnaire  and 
responded.  Additionally,  other  F-22  government  managers  not  directly  supporting  Spiral 
2  or  modernization  were  asked  to  respond  to  the  questionnaire.  All  government 
personnel  who  were  provided  the  questionnaire  were  assigned  to  the  F-22  System 
Program  Office  (SPO). 
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2.  Summary  of  Responses 

There  were  25  responses  to  the  questionnaire  from  both  government  and 
contraetor  F-22  personnel. 30  Seventeen  (68  pereent)  of  the  respondents  were  government 
SPO  personnel.  Although  these  17  respondents  only  represent  approximately  9  pereent 
of  the  personnel  assigned  to  the  F-22  program  office  at  the  time  of  the  studySi,  they 
accounted  for  100  percent  of  government  personnel  assigned  to  the  Spiral  2  program. 
Additionally,  all  eight  contractor  respondents  were  Spiral  2  personnel.  While  the  sample 
for  the  questionnaire  was  well-below  a  meaningful  representation  for  the  entire  F-22 
program,  it  was  a  very  good  representation  for  the  Spiral  2  program.  Additionally,  when 
looking  at  the  area  of  the  program  supported  by  the  respondents,  22  of  the  25  responses 
(88  percent)  were  from  personnel  who  support  the  modernization  program.  Because  of 
the  functional  matrix  organizational  approach  used  by  the  F-22  program,  it  was  difficult 
to  determine  the  total  number  of  personnel  working  on  the  modernization  program; 
however,  it  was  strongly  believed  by  the  authors  that  these  22  individuals  represented  a 
statistically  significant  number  of  modernization  personnel. 

Figure  13  shows  how  the  respondents’  support  of  program  areas  and  elements  was 
distributed  for  government  and  contractor  personnel.  When  categorizing  the  program 
element(s)  on  which  the  respondents  worked  there  was  a  majority  of  respondents  who 
supported  avionics.  Because  personnel  support  multiple  areas  and  elements  of  the 
program,  respondents  were  permitted  to  select  more  than  one  functional  area  and 
program  element. 


30  One  response  was  deemed  invalid  due  to  incomplete  answers. 

31  Based  on  F-22  System  Program  Office  headcount  of  194  active  duty  military  and 
DoD  civilians  provided  in  telephone  interview  with  front  office  personnel  on  13 
November  2006. 
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Figure  13.  Distribution  of  Respondents 


When  looking  at  some  of  the  speeifie  responses  from  the  questionnaire,  there 
were  definite  trends  in  some  areas.  Some  of  the  most  notable  of  these  were  with  respect 
to  the  32  ANSI/EIA  criteria.  Nineteen  (76  percent)  of  the  respondents  stated  they  were 
not  aware  of  the  32  criteria.  Considering  the  importance  of  these  criteria  in  determining 
whether  an  EVMS  has  been  correctly  implemented,  it  was  surprising  to  see  how  many 
users  were  unaware  of  the  criteria.  Perhaps  this  was  why  EVM  could  have  been 
implemented  on  Spiral  2  and  had  as  many  criteria  be  judged  as  marginal  in  their 
implementation.  Another  interesting  trend  with  respect  to  the  32  criteria  was,  of  the  six 
respondents  who  said  they  were  aware  of  the  32  criteria,  all  stated  they  believed  EVM 
was  being  implemented  on  E-22  in  compliance  with  these  criteria.  The  binary  (Yes/No) 
response  to  the  question  “Do  you  think  your  program  implements  its  EVMS  according  to 
the  32  ANSEElA-748  Earned  Value  Management  System  criteria?”  may  have  forced 
some  respondents  to  go  with  the  “predominate”  response — after  all,  the  question  didn’t 
ask  if  their  program  EVMS  meets  all  of  the  32  criteria.  It  was  still  surprising  for  the 
authors  to  see  100  percent  of  these  respondents  respond  in  the  affirmative  to  this  question 
considering  the  authors’  findings  on  the  criteria  were  not  unanimous. 
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Additional  trends  were  observed  with  respeet  to  contraetor  versus  government 
responses.  The  most  notable  was  the  inereased  frequeney  of  EVM  use  on  the  part  of 
contractor  respondents  versus  government  respondents  (see  Figure  14).  Of  the  17 
government  personnel  who  stated  some  use  of  EVM,  15  (88  percent)  of  the  respondents 
stated  they  used  EVM  less  than  once  per  week,  while  two  stated  they  used  it  once  per 
week.  Contrasting  this  to  the  seven  (87.5  percent)  of  the  eight  contractor  respondents 
who  all  stated  they  used  EVM  at  least  once  per  week.  Of  these  seven,  two  (25  percent)  of 
the  contractors  stated  they  used  EVM  more  than  once  per  week.  This  disparity  in 
frequency  of  use  was  not  a  surprise  to  the  authors  based  on  the  concern  raised  by 
government  personnel  regarding  the  timeliness  of  EVM  data.  As  discussed  above,  while 
contractor  managers  have  access  to  preliminary  EVM  data  less  than  one  week  old,  the 
government  only  receives  EVM  data  once  it  is  provided  in  formal  reports  that  are  weeks 
later  than  the  work  being  reported.  This  disparity  in  timeliness  is  the  most  likely  driver 
for  the  disparity  in  frequency  of  use  between  the  two  organizations. 


Never  Once  per  Month  More  than  Once  Once  per  Week  More  than  Once 

per  Month  per  Week 


Frequency  of  EVM  Use 


Figure  14.  Frequency  of  EVM  Use  by  Government  and  Contractor  Respondents 
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3,  Propositions 

The  process  of  evaluating  the  responses  to  the  questionnaires  resulted  in  several 
propositions,  each  of  which  enabled  further  analysis  of  potential  relationships  and  trends 
regarding  EVMS  usage  and  perceived  value.  Each  of  these  propositions  was  evaluated 
individually  using  questionnaire  responses  to  determine  their  validity.  Although  the 
results  of  the  analysis  of  propositions  are  presented  below,  more  detailed  supporting  data 
and  values  for  the  questionnaire  responses  and  analysis  is  provided  in  Appendix  III. 


a.  Higher  EVM  Value  Results  in  More  EVM  Usage 

The  first  proposition  was  that  an  individual  who  places  higher  value  on 
EVMS  as  a  management  tool  will  use  EVMS  more  frequently.  Of  the  25  respondents,  17 
(68  percent)  stated  they  viewed  EVMS  as  having  moderate  or  lower  value  (three  or  less 
on  a  scale  of  one  to  five).  Meanwhile,  there  were  16  (64  percent)  respondents  who  stated 
they  used  EVMS  more  than  once  per  month.  When  testing  the  relationship  between  these 
two  variables,  there  was  not  enough  statistical  evidence  to  accept  this  proposition  with  95 
percent  certainty.  However,  there  was  enough  of  a  relationship  between  these  variables 
to  state  that,  having  90  percent  certainty,  the  more  value  an  individual  placed  on  EVMS, 
the  more  they  used  the  tool. 

This  proposition,  supported  by  the  questionnaire  results32,  provided  some 
measure  of  confidence  in  the  validity  of  the  questionnaire.  It  would  have  been  counter¬ 
intuitive  for  there  to  not  have  been  a  relationship  between  perceived  value  and  frequency 
of  use.  Any  tool  that  is  truly  believed  to  be  valuable  should  be  used  more  often. 

b.  Higher  EVM  Usefulness  Results  in  More  Usage  of  EVM  in 
Managing 

The  second  proposition  was  that  individuals  who  had  an  opinion  of  higher 
usefulness  for  EVMS  would  use  it  more  as  a  management  tool  versus  just  a  reporting 

32  Per  statistical  analysis,  propositions  are  either  “rejected”  or  “not  rejected.”  Eailure 
to  reject  a  proposition  does  not  necessarily  make  it  “true.”  Eor  simplicity,  the  authors 
refer  to  propositions  that  were  “not  rejected”  as  being  true. 
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tool.  Although  EVMS  is  intended  to  be  used  as  a  management  tool,  1 1  (44  percent)  of 
the  F-22  respondents  either  solely  or  predominately  used  EVMS  to  receive  and  report 
cost  and  schedule  status.  Twelve  (48  percent)  of  the  respondents  stated  they  used  EVM 
data  either  equally  as  a  management  and  reporting  tool  or  predominately  as  a 
management  tool. 33  As  discussed  above,  many  times  the  timeliness  of  the  EVM  reports 
limited  the  utility  of  EVM  as  a  management  tool.  This  proposition  sought  to  identify  any 
linkage  between  respondents’  perceived  usefulness  and  more  meaningful  usage  of  the 
data.  This  proposition  was  also  supported  by  the  responses  with  90  percent  certainty. 

The  difficulty  in  interpreting  this  proposition  was  which  variable  was  the 
driver.  That  is,  did  method  of  use  drive  perceived  value  (i.e.,  a  management  tool  is  likely 
to  be  deemed  more  “useful”  than  a  reporting  tool)  or  did  perceived  usefulness  drive 
method  of  use  (i.e.,  a  meaningful  source  of  data  will  be  used  for  managing  instead  of 
merely  reporting)?  This  question  could  not  be  answered  by  the  data  collected  in  the 
questionnaire;  however,  it  was  important  enough  merely  to  confirm  there  was  a  linkage 
between  these  two  responses.  Regardless  of  the  driver,  the  relationship  existed  in  the 
questionnaire  responses  and  pointed  to  the  important — perhaps  essential! — linkage 
between  the  two  variables. 


c.  Higher  EVM  Knowledge  Results  in  Higher  EVM  Value 

The  final  proposition  was  that  individuals  who  identified  themselves  as 
having  more  knowledge  regarding  EVM  would  place  a  higher  value  on  EVM.  The 
rationale  behind  this  proposition  was  that  individuals  who  knew  more  about  the  data  and 
application  of  EVM  would  place  more  value  on  its  ability  to  be  used  as  a  management 
tool.  Based  on  the  results  of  the  questionnaire,  however,  this  was  not  the  case. 
Regardless  of  the  level  of  certainty,  there  was  nothing  in  the  results  to  support  this 
proposition.  This  indicated  respondents  determine  their  value  in  EVMS  via  other  means. 
One  likely  source  would  be  their  own  “return”  on  using  the  EVM  products.  This  is  just 
one  potential  driver  and  the  question  of  where  the  value  was  derived  was  not  specifically 

33  Three  respondents  did  not  identify  any  method  of  use  in  their  questionnaire 
responses. 
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addressed  in  the  questionnaire.  If  this  questionnaire  is  representative,  it  does  indieate  that 
merely  teaehing  and  training  about  EVM  is  not  suffieient  to  eonvince  managers  to  value 
the  tool. 

While  the  questionnaire  results  eould  be  broken  down  and  analyzed  in  any 
number  of  additional  ways,  the  authors  believe  these  propositions  are  the  most  relevant  to 
answer  the  question  of  whether  EVM  is  used  by  E-22  personnel  to  manage  the  program. 
The  following  final  ehapter  will  address  the  authors’  findings  regarding  this  and  other 
questions  posed  in  this  paper. 


E,  SUMMARY 

This  eoneludes  Chapter  IV,  F-22  EVMS  Environment.  The  next  ehapter. 
Conclusion,  presents  a  condensed  synopsis  of  this  research  project’s  outcome,  includes  a 
brief  discussion  on  limitations  with  respect  to  the  research  project,  and  makes  final 
recommendations  to  the  EVMS  where  necessary. 
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V.  CONCLUSION 


A.  SUMMARY  OF  FINDINGS 

The  intent  of  this  study  was  to  academically  appraise  the  F-22  program’s  use  of 
EVM  in  managing  avionics  software  development  within  the  Spiral  2  REDI  contract 
effort,  Delivery  Orders  0002  and  0019.  Achieving  this  goal  involved  performing  and 
reporting  the  results  of  1 .)  a  detailed,  data-supported  evaluation  of  how  the  program 
meets  each  of  the  32  ANSI/EIA  EVMS  criteria;  2.)  interviews  with  subject-matter 
experts;  3.)  a  statistical  questionnaire  conducted  with  government  and  contractor 
personnel  involved  in  E-22  software  development.  In  order  to  facilitate  the  assessment, 
there  were  three  questions  asked  by  the  authors.  The  following  are  the  findings  and 
recommendations  associated  with  each  question: 

1.  How  Closely  Did  Implementation  of  EVM  Follow  ANSI/EIA  Criteria? 

a.  Findings 

Answering  “How  closely  did  the  F-22  Spiral  2  implementation  of  EVM 
follow  the  criteria  outlined  in  ANSI/EIA-748-A-1998  Earned  Value  Management 
System?”  relied  mainly  on  a  careful  assessment  of  how  EVM  implementation  for  Spiral  2 
supported  the  32  ANSI  criteria.  Based  on  the  review  of  the  objective  of  each  criterion, 
the  F-22  self-described  implementation  of  the  criteria,  and  interviews,  data,  and  direct 
observation  an  assessment  was  made  for  each  of  the  32  criterion.  These  assessments 
were  solely  those  of  the  authors  and  do  not  represent  an  official  government  position  on 
the  F-22  implementation  of  their  EVMS.  A  summary  of  the  32  criteria  assessments  are 
as  follows:  seven  excellent  criteria,  15  satisfactory  criteria,  five  marginal  criteria,  and  one 
insufficient  criterion,  and  four  inconclusive  criteria.  With  22  (69  percent)  of  the  32 
criteria  assessed  as  satisfactory  of  better,  the  F-22  implementation  of  EVM  was  fairly 
strong;  however,  improvement  of  the  F-22 — and  follow-on  programs’ — use  of  EVM 
during  software  development  strongly  depends  on  a  discussion  of  the  marginal  and 
insufficient  criteria. 


99 


Although  there  were  five  criteria  deemed  marginal,  three  of  the  five 
criteria  were  in  one  of  the  five  criteria  groups.  Three  of  the  ten  Planning,  Scheduling  and 
Budgeting  criteria  were  marginal.  Another  criterion  could  not  be  assessed  sufficiently  for 
a  rating  due  to  lack  of  supporting  data.  This  trend  was  consistent  with  discussions  and 
direct  observations  regarding  the  environment  of  F-22  development.  As  discussed  in 
Chapter  IV,  F-22  EVMS  Environment,  the  use  of  undefinitized  contracts  and 
requirements  instability  made  it  extremely  difficult  to  adequately  plan,  schedule  and 
budget  for  Spiral  2  efforts. 

Criterion  #26,  Implement  managerial  actions,  was  the  only  criterion  to  be 
assessed  as  insufficient.  This  was  largely  based  on  the  apparent  absence  of  managerial 
actions  taken  on  DO  0002  even  as  it  had  excessive  variances  and  progressed  toward  a 
significant  cost  underrun.  While  it  seems  counter-intuitive  to  criticize  a  cost  underrun,  if 
corrective  measures  had  been  taken  earlier,  the  program  may  have  been  able  to  capitalize 
on  these  additional  funds  before  DO  0002  work  was  completed.  The  potential  reasons 
for  the  failure  of  the  program  to  take  managerial  actions  are  discussed  in  the  following 
section.  Suffice  it  to  say,  it  does  little  good  to  collect  EVM  data  if  the  program  chooses 
not  to  use  it. 


b.  Recommendations 

If  EVM  is  to  be  successfully  used  by  E-22  personnel  to  manage  avionics 
development,  research  data  suggests  that  program  managers  dedicate  themselves  towards 
moving  away  from  the  use  of  undefinitized  contracts.  While  this  is  easier  stated  than  it  is 
accomplished,  based  on  the  goals  and  intent  of  many  of  the  ANSI/EIA  EVMS  Criteria, 
EVMS  will  not  be  fully  useful  as  a  management  tool  to  E-22  as  long  as  it  is  used  in 
concert  with  undefinitized  contracts. 

A  process  for  ensuring  more  disciplined  control  of  requirements  should  be 
sought  after.  Research  showed  that  stable  requirements  are  critical  for  reliable,  maturing 
EVM  products.  Group  2  of  the  ANSI/EIA  EVMS  Criteria  focus  on  Planning,  Scheduling 
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and  Budgeting.  It  is  difficult  if  not  impossible  to  meet  the  intent  of  the  criteria  outlined 
under  this  area  if  requirements  are  not  well-defined. 

2,  To  What  Degree  Did  Implementation  of  EVM  Fulfill  Its  Management 

Control  Role? 

a.  Findings 

In  Chapter  II,  Literature  Review,  there  were  historical  indicators  that  the 
F-22  program  either  chose  to  ignore  or  did  not  have  confidence  in  the  message  that  EVM 
was  providing  during  EMD.  After  collecting  and  analyzing  Spiral  2  CPR  data  along  with 
assessing  responses  to  the  EVM  Questionnaire,  the  authors’  conclusion  was  there  are  still 
problems  within  the  E-22  program  in  this  area.  One  driver  for  the  lack  of  use  of  EVM  as 
a  management  tool  is  lack  of  confidence  in  EVM.  It  could  not  be  determined  whether 
this  was  due  to  unique  E-22  problems  (e.g.,  rolling  and/or  immature  PMB)  or  to  a  more 
general  lack  of  perceived  EVM  value.  Both  questionnaire  results  and  interviews  did 
indicate  less  use  of  EVM  on  the  part  of  the  government  compared  to  the  contractor.  This 
could  have  been  due  to  the  significant  time  delay  between  when  contractor  managers  first 
see  EVM  data  and  when  government  managers  finally  see  it. 

When  considering  managerial  tools  for  use,  the  timeliness  of  the  data  is 
critical.  However,  the  bottleneck  in  this  instance  appeared  to  be  EM  Aero  business 
practices,  not  the  EVMS.  Streamlining  these  business  practices  should  be  considered 
paramount  when  attempting  to  shorten  the  timeline  between  EVM  data  collection  and 
reporting. 

The  analysis  of  Spiral  2  EVM  reports  did  indicate  some  errors  in 
accounting/reporting;  however,  none  of  these  attained  a  magnitude  sufficient  enough  to 
bring  into  question  the  usefulness  of  the  EVM  reports.  While  there  should  be  questions 
asked  regarding  these  errors  to  ensure  accurate,  trustworthy  reporting,  the  authors  did  not 
believe  these  errors  were  systemic  in  nature.  The  Spiral  2  data,  like  the  EMD  data,  had  a 
story  to  tell.  As  discussed  above,  there  was  a  lack  of  action  in  response  to  this  data.  This 
did  not  appear  to  be  a  result  of  some  malfunction  of  the  E-22  EVMS,  but  rather  inaction 
on  the  part  of  program  managers. 
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b. 


Recommendations 


If  EVM  is  to  fulfill  its  role  as  a  managerial  tool  in  the  F-22  program, 
research  concludes  that  steps  should  be  taken  to  ensure  government  access  to  EVM  data 
in  a  timelier  manner.  Options  should  be  explored  and  steps  taken  to  leverage  current 
customer-oriented  industry  practices  that  employ  business  procedures  to  shorten  response 
timelines  (including  customer  reports).  Data  suggests  that  investment  into  such  process 
optimization  will  allow  a  more  complete  utilization  of  any  EVMS. 

3.  To  What  Extent  Did  F-22  Management  Use  EVM  to  Manage  Avionics 

Development? 

a.  Findings 

The  EVM  Questionnaire  aimed  to  assess  the  perceived  usefulness  of  EVM 
within  a  software  development  context  and  to  assess  E-22  managers’  perceived  value  of 
EVM  with  respect  to  their  management  duties.  Interviews  supplemented  the  data  from 
the  questionnaire,  figuratively  filling  in  the  questionnaire’s  information  gaps  uncovered 
throughout  the  course  of  the  research  project.  Combining  these  assessment  tools  with 
those  used  for  the  other  areas  of  research  provided  a  full  picture  of  how  E-22  avionics 
managers  use  EVM.  Questionnaire  results  indicated  there  is  a  somewhat  low  perceived 
value  for  EVM.  Additionally,  there  were  indications  that  EVM  is  used  more  as  a  tool  for 
receiving  and  reporting  cost  and  schedule  status  versus  using  it  as  a  tool  for  proactively 
managing  the  development  effort.  Based  on  interviews  conducted  with  several  E-22 
avionics  managers,  this  becomes  truer  as  software  development  efforts  progress  from 
design  towards  coding  and  into  integration  and  testing.  The  more  defined  the  tasks,  the 
more  confidence  existed  in  EVM  reports.  As  programs  such  as  Spiral  2  moved  toward 
less  defined  tasks,  the  less  value  was  placed  on  EVM  and  the  less  it  was  used  as  a 
management  tool. 
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b.  Recommendations 

Results  of  the  data  analysis  and  questionnaire  analysis  indicate  a  clear 
trend  of  F-22  managers  either  ignoring  EVMS  data  or  not  using  it  enough  for  it  to  be 
considered  an  active  management  tool.  The  authors  submit  this  is  largely  due  to  the 
perceived  value  of  F-22  EVMS  products.  Regardless  of  whether  it’s  driven  by  the 
managers’  knowledge  of  F-22  EVMS  flaws  (as  pointed  out  in  this  paper)  or  by  a  more 
generic  lack  of  appreciation  for  EVMS  as  a  management  tool,  if  the  F-22  software 
development  programs  are  going  to  take  advantage  of  the  benefits  of  EVMS,  the  program 
must  first  establish  confidence  in  it.  Some  of  this  confidence  can  be  gained  by 
implementing  earlier  recommendations  that  would  make  available  more  reliable,  stable 
and  timely  data  to  government  and  contractor  managers.  Additional  confidence  could  be 
gained  by  providing  focused  training  for  software  development  managers  on  the  strengths 
and  weaknesses  of  EVMS  with  respect  to  software  development.  This  would  enable 
managers  to  become  more  knowledgeable  regarding  how  program  decisions  (such  as 
proceeding  with  UCAs)  affect  the  value  and  usefulness  of  EVM. 

EVMS  is  particularly  useful  in  the  early  stages  of  software  development; 
however,  research  indicates  it  becomes  less  reliable  and  useful  as  the  software 
development  program  progress  through  integration  and  test.  Although  a  suggestion  of  a 
another  software  management  tool  is  beyond  the  scope  of  this  research  project,  the  F-22 
program  should  leverage  off  of  knowledgeable  resources  such  as  the  Carnegie  Mellon 
Software  Engineering  Institute  and  the  USAF  Software  Technology  Support  Center  to 
explore  the  use  of  other  management  tools  to  either  augment  or  replace  EVM  during  the 
latter  stages  of  software  development. 

4,  Summary 

The  cumulative  effect  of  assessing  the  criteria,  evaluating  the  Spiral  2  EVM  data 
and  assessing  questionnaire  responses  along  with  interview  statements  indicated  that  in 
the  case  of  Spiral  2,  there  is  a  limited  use  of  EVM  by  F-22  personnel  in  managing 
avionics  development  efforts.  It  is  necessary  for  the  program  to  evaluate  the  reasons 
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behind  this  conclusion,  whether  it  was  lack  of  perceived  value,  lack  of  understanding  of 
its  function,  or  lack  of  confidence  in  the  data.  Regardless  of  the  reason,  data  suggests 
that  the  F-22  program  management  should  either  take  steps  to  address  better  use  of  EVM 
or  identify  other  management  tools  in  its  stead.  Either  of  these  actions  will  help  assure 
the  program  avoids  repeating  performance  challenges  endured  during  EMD. 

B,  LIMITATIONS 

Within  the  context  of  this  project,  every  effort  had  been  made  towards  a  full 
assessment  of  the  E-22  avionics  program’s  EVM  implementation,  replete  with  all 
essential  information.  Unfortunately,  reality  dictated  something  less  than  full  collection 
of  required  data.  Regardless  of  completeness,  the  very  information  collected  during  the 
course  of  this  academic  pursuit  may  represent  an  outcome  approximating,  rather  than 
equal  to,  the  reality  of  the  situation.  As  such,  several  limiting  factors  with  regard  to  this 
particular  study  must,  in  the  name  of  full  academic  disclosure,  come  to  light. 

To  begin  with,  the  nature  of  cutting-edge  weapons  systems  like  the  E-22  dictated 
that  certain  information  remained,  in  some  cases,  privileged.  Thus,  during  the  course  of 
this  study’s  investigative  phase  some  information  collected  was  accompanied  with 
instructions  to  disseminate  in  a  general  manner,  as  opposed  to  reporting  details  that  may 
or  may  not  have  altered  this  project’s  outcome  or  its  readers’  opinions.  This  requirement 
also  pertained  to  first-hand  knowledge  that  the  authors  may  or  may  not  have  added  to  this 
paper’s  findings. 

Secondly,  and  in  a  much-related  matter,  the  nature  of  government  and  defense 
contractor  relations  necessitated  a  certain  amount  of  caution  during  the  course  of 
interactions  between  representatives  of  the  two  parties.  Unfortunately,  though  both 
parties  make  sincere  efforts  to  maintain  a  mutually  beneficial  environment  of  information 
sharing,  in  the  past  both  parties  have  failed  to  ensure  that  the  sharing  of  sensitive 
information  (e.g..  Government  Budgets,  “Insider”  schedule  and  cost  reports)  does  not 
harm  the  party  sharing  that  potentially  inflammatory  data.  As  a  result,  this  project’s 
investigative  efforts  fell  short  of  acquiring  some  data  initially  requested  from  the 
contractor. 
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Finally,  the  analysis  phase  of  this  projeet  has  uncovered  issues  regarding  the 
EVM  Questionnaire.  The  first  issue  dealt  with  the  small  and  specific  population  size  that 
the  questionnaire  analysis  depends  upon.  This  introduced  biases  rooted  in  program 
culture  and  program-specific  training,  even  within  the  context  of  the  F-22  SPO  /  LM 
Aeronautics:  F-22.  Thus,  should  problem  areas  with  respect  to  EVM  implementation 
become  visible,  an  analysis  of  questionnaire  data  may  or  may  not  uncover  systemic 
causalities  that  “perfect”  information  might  identify. 

The  second  questionnaire  issue  involved  the  discrete  values  assigned  to  the  EVM 
Questionnaire  queries.  Eor  example.  Question  #3’s  possible  answer  began  with  “Never” 
and  proceeds  to  “Once  a  Month”.  Post-submission  analysis  indicated  that  a  substantial 
number  of  personnel,  especially  those  not  identifying  “Einancial  Management”  as  their 
“current  role”,  actually  would  answer  somewhere  in-between  these  given  choices  (i.e., 
more  than  “Never”  but  less  than  “Once  a  Month”).  Thus,  though  careful  thought  went 
into  the  questionnaire  design  and  implementation,  the  discrete  choices  given  to 
respondents  appeared  to  have  affected  the  questionnaire  analysis. 

Eimitations  aside,  the  information  gathered  and  interpreted  provided  sound  data 
collection,  results,  and  recommendations  given  realistic  constraints  that  would  affect  any 
similar  undertaking.  An  honest  appraisal  of  research  limitations  should  not  preclude  the 
fact  that  perfect  information  almost  never  avails  itself  to  those  who  seek  it.  That  said,  the 
remainder  of  this  chapter  reviews  the  highlights  of  this  study  and  presents 
recommendations  for  consideration. 

C.  AREAS  FOR  FUTURE  STUDIES 

Although  the  authors  sought  to  extensively  assess  the  use  of  EVM  in  the  E-22 
program,  several  areas  for  further  study  were  identified  on  this  topic.  The  first  was  in  the 
area  of  applicability  of  EVM  to  the  latter  stages  of  software  development.  As  software 
development  progresses  from  well-defined  tasks  such  as  development  of  design 
documents,  writing  code  and  developing  test  plans  to  less  defined  tasks  such  as  recode 
and  retest,  EVM  becomes  less  useful  as  a  management  tool.  Research  in  the  area  of 
potential  approaches  for  measuring  value  in  this  less-defined  regime  would  be  valuable. 
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Spiral  2  has  not  yet  completed  its  DO  0019  effort  which  is  largely  comprised  of  these 
“less-defined”  tasks.  A  study  of  how  EVM  implementation  was  employed  at  the 
conclusion  of  this  delivery  order  would  be  useful  in  identifying  potential  alternatives  or 
variations  to  current  EVM  implementation. 

E-22  is  plagued  with  reliance  on  undefinitized  contractual  actions.  Considering 
many  of  the  challenges  facing  E-22  are  not  unique,  it  is  reasonable  to  assume  excessive 
use  of  UCAs  is  also  not  unique.  Research  into  the  trends  and  impacts  of  this  type  of 
contractual  approach  might  provide  further  information  to  managers  and  contract 
officers.  UCAs  are  not  wrong;  however,  their  use  does  come  with  consequences. 
Research  that  quantified  these  consequences  would  be  useful. 

Another  challenge  identified  for  the  E-22  program  was  the  timeliness  of  EVM 
reports  to  government  managers.  This  is  also  not  an  issue  unique  to  the  E-22  program. 
As  the  authors  discussed,  there  are  valid  reasons  behind  the  delay  in  providing  trusted 
EVM  data  to  the  government;  however,  there  are  potential  solutions  to  these 
challenges. 34  Research  into  the  major  drivers  for  this  reporting  delay,  potential  solutions, 
and  costs  associated  with  these  solutions  could  provide  DoD  some  viable  options  to  solve 
this  issue  and  make  EVM  as  valuable  to  the  government  as  it  is  to  the  contractor. 


34  See  Recommendations  for  Research  Question  #2. 
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SPIRAL  2  PHASE  B:  EV  DATA  _ 

Work  Prior  to  Initial  Reporting  Period  (Pre-April  2004) 
(000s) 


APPENDIX  A  -  SPIRAL  2  DO  0002  COST  AND  EVM  DATA 
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Data  lacks  verification  past  whole-number  values;  thus,  _  Contract  Budget  Ba$e  46,043 
rounding  errors  have  occurred.  Delta 
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APPENDIX  B  -  EVM  QUESTIONNAIRE 


Please  answer  the  followins  questions  by  circlins  your  selection. 


QUESTION  #1  Using  the  options  provided  below,  eircle  the  function  that  best 
describes  your  current  role  within  the  F-22  program: 

Government  Employee 


Contract  p  Financial  Program 

Manager  ngineer  Manager  Manager 


Other 


Contractor  Employee 


Contract  p  Financial  Program 

Manager  ngineer  Manager  Manager 


Other 


Other: 


QUESTION  #2  Using  the  options  provided  below,  identify  the  areas  within  the  F-22 
program  you  have  working  knowledge  of  (circle  all  that  apply): 


Airframe 


Avionics  Support  propulsion 
Systems 


Other 


Other: 


EMD  Modernization  Production  Sustainment  Other 


Other: 


QUESTION  #3  Using  the  scale  provided  below,  identify  how  often  you  use  or  are 
exposed  to  some  form  of  E VMS: 


Never 


Once  per 
Month 


More  Than 
Once  per 
Month 


Once  per 
Week 


More  Than 
Once  per 
Week 
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QUESTION  #4  Using  the  options  provided  below,  identify  whieh  best  deseribes  how 
you  utilize  your  program’s  EVM  information: 


Reporting  Tool  Equal  Parts  Management 
Reporting  with  (some)  Reporting  and  Tool  with  Management 

Tool  Only  Management  Management  (some)  Tool  Only 

Uses  Tool  Reporting  Uses 

QUESTION  #5  Using  the  options  provided  below,  please  assess  the  value  of  your 
program’s  EVM  information: 

1  2  3  4  5 


Very  Eittle  Value 


- ► 

Very  High  Value 


QUESTION  #6  Using  the  seale  provided  below,  please  self-assess  your  knowledge  of 
EVM: 


1  2  3  4  5 


What’s  EVM,  again? 


- ► 

I  am  an  EVM  master! 


QUESTION  #7  Are  you  aware  of  the  32  eriteria  set  forth  by  ANSI/EIA-748  for 
assessing  Earned  Value  Management  Systems? 

YES  NO 

If  you  answered  “YES”,  please  proeeed  to  question  #8. 

If  you  answered  “NO”,  please  proeeed  to  question  #9. 


QUESTION  #8  Do  you  think  your  program  implements  its  EVMS  aeeording  to  the  32 
ANSI/EIA-748  Earned  Value  Management  System  eriteria? 

YES  NO 


QUESTION  #9  Using  the  options  provided  below,  please  deseribe  the  usefulness  of 
your  program’s  EVMS: 


1  2  3  4  5 


Not  at  All  Useful 


- ► 

Very  Useful 
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APPENDIX  C  -  EVM  QUESTIONNAIRE  RESULTS 


A,  RAW  QUESTIONNAIRE  RESPONSE  DATA 
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21 
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1 

: 
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3 
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B.  QUESTIONNAIRE  STATISTICAL  ANALYSIS  RESULTS 
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